Reprocessing data with XDS at home

Windows Users

Before you start there are a few things you need to do:

• Make a folder in your C drive where your data will go. You can call it whatever you like. In this example, I have called it 'reprocess' 

• Download the data you want to reprocess from the synchrotron server (follow these instructions). You will need the .h5 files and the XDS.INP files. 

  • The .h5 files can be found under /data/EPN/data/frames/user/crystal.

  • The XDS.INP file can be found under /data/EPN/data/home/user/auto/dataset/crystal

  • Where EPN is the EPN of your experiment, user is the user folder with your data, and crystal is the name of the dataset

• Put the data you wish to reprocess in the new folder on your C drive. See Step 1 of Running XDS below for how to arrange files. 

Important note, the C drive is accessed on Ubuntu as:

Install Ubuntu

Installing WSL 1 (Windows Subsystem for Linux)

1. Right click on Windows PowerShell app and select “Run as Administrator”

Open

2. Type and run the following command (all one line, be careful to include spaces in correct places):

3. Go to the Microsoft Store and select the Linux distribution you want to run. In this case, Ubuntu. Click 'Get' to install, then open the newly installed distribution. 

Open

4. Create a user account and password for your Linux distribution. You only need to do this the first time you launch.

If Ubuntu is not working try restarting your computer.

Install XDS

1.Open Ubuntu and run the following commands:

In plain English:

Line 1 – change directory to the place binary files are stored (“binary file” is just a name for a non-text file)

Line 2 – use the internet to get the XDS files (compressed) from the website

Line 3 – make a softlink from the long XDS-INTEL-blahblah path name to “here” (period means “here” in Linux). In this case, “here” is /usr/local/bin

If there is an error when running the second line of code, try going into super user mode by running

2. Check the permissions of your XDS programs and ensure they’re “executable”. 

Do this by running:

If the file permissions are not as shown in above picture, you need to modify them to make them executable by running:

3. Confirm XDS has installed correctly by running 

which should return:

And check if xds_par has installed correctly by running:

which should return:

XDS is now ready to use!

After XDS is installed you should exit super user mode, which can be done by running:

dectris-neggia library

There are some additional tools you will need before being able to process images from the Australian Synchrotron. The first of which is the ‘dectris neggia library’. This is essentially a file that tells XDS how to read the .h5 files output from the current Eiger detectors. 

The file you need is called dectris-neggia.so. Download from link below. 

Open

Once you have downloaded it, make sure it is somewhere on your C drive, and then you will need to copy it to a more useful location. 

Important note, the C drive is accessed on Ubuntu as:

Open Ubuntu and copy the dectris-neggia.so file to /usr/local/bin

To copy any file in linux you run a command like this:

If you get an error saying 'permission denied' then add sudo to the start of the line. 

For this example, I ran:

You will need to edit this line to account for the correct path on your C drive for the location of the dectris-neggia.so file. 

Running XDS

Important note, the C drive is accessed on Ubuntu as:

1.Arrange your folders and files

In the reprocessing folder in your C drive, you will need to make a new folder for each dataset you wish to reprocess. 

In the folder add the XDS.INP file, and then make a new folder called 'img' and add the .hf files. 

eg, for a dataset called RuBy the folders should look like this

2. Edit XDS.INP file

Check the XDS.INP file to see if it is going to run what you want. 

Information about each of the entries of the XDS.INP file can be found on the via the XDS wiki

http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html

The main entries to check will be:

JOB= 

which refers to which xds processes you want to run. If you wish to reprocess your data from scratch, then it should read:

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

LIB=

which is where it will look for the dectris-neggia.so file. If you have followed the above instructions, it should be:

LIB= /usr/local/bin/dectris-neggia.so

NAME_TEMPLATE_OF_DATA_FRAMES= 

This is where it will look for the frames. If you have put your .h5 files in a folder called 'img' inside your dataset folder, then it should read something like:

NAME_TEMPLATE_OF_DATA_FRAMES= img/RuBy_1s_0065_??????.h5

for a dataset which was called 'RuBy_1s_0065'. The ?????? should be there!

MAXIMUM_NUMBER_OF_PROCESSORS=

This is how many processors XDS will use on your computer to run. You probably want to set this to 2 less than the amount your computer has. 

To check how many you have on your computer (on Windows), Press Ctrl + Shift + Esc to open Task Manager. Select the Performance tab to see how many cores and logical processors your PC has. 

3. Run XDS

In Ubuntu, cd to your dataset folder (the one with the XDS.INP file and the img folder) and run xds by running:

If you have assigned more than one processor to run xds then you can run xds_par (which is xds parallel)

If xds is running correctly, you will see a log file being written in your terminal, and new files being written in your dataset folder. 

Mac Users

Installing XDS

Instructions taken from https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation:

Attention: generate_XDS.INP requires the Xcode command line tools to be installed (free of charge): open a Terminal window, and enter sudo xcode-select --install. You may have to explicitly agree to the License terms when running a Command Line Tool (e.g. strings) for the first time.

Become familiar with the concept and ways to run commands as "root" - google "mac osx become root". All the installation tasks that are run in a Terminal window require root privileges, since some of the programs and their links are written to /usr/local/bin. (As an alternative that does not require root for installation, one may create a directory $HOME/bin and use that for the programs and links. That would also require modification of the $PATH, by a one-time echo 'export PATH=$PATH:$HOME/bin' >> ~/.profile)

So, prepare the following steps by opening the Terminal window, and then

sudo su                    # this will make you root, and ask for your password

mkdir /usr/local/bin  # only if /usr/local/bin was not created before

XDS package

If you are an academic user,

echo you may have to "sudo su" first, to obtain administrator permissions!

cd /usr/local/bin

curl -L -o - ftp://ftp.mpimf-heidelberg.mpg.de/pub/kabsch/XDS-OSX_64.tar.gz | tar xzvf -

ln -sf XDS-OSX_64/* .