Processing DAC data with CrysalisPro

Owned by
MX Cluster (Unlicensed)
Last updated: Jun 24, 2024 by
AS MX Wiki
8 min read

At the Synchrotron

1. Download frames

On the MX windows CrysalisPro processing box, log in via these credientials:

User: mxuser

Password: Beam109Line

via filezilla or similar program, download the frames of the experiment. Instructions

They are in .h5 files, each of which contains 200 eiger frames (up to 20 degrees). 

/data/mx/EPN/frames/$username/d

where $EPN and $username are the experiment number and the user name, respectively. 

You will need each of the .h5 files, including the master file, for each experiment. 

Make a new folder for each dataset you are processing. If you want to process more than one wedge of data at a time, put all of the frames in the same folder.

At Home

1. Download frames

Via filezilla or similar program, download the frames of the experiment. Instructions

Frames in folder: /data/mx/EPN/frames/$username/d

where $EPN and $username are the experiment number and the user name, respectively. 

Frames are in .h5 files, each of which contains 200 eiger frames (20 degrees). 

You will need each of the .h5 files, including the master file, for each experiment. 

It is a good idea to make a new folder for each collection, if you didn't do this during data collection.  If you want to process more than one wedge of data at a time, put all of those frames in the same folder. Commonly you will want to process together runs 1 and 2, 3 and 4, and 5 and 6. So you likely want to make 3 folders, each containing a pair of runs.

2. Convert h5 files to cbf using eiger2cbf (on windows computer at MX1)

  • On the desktop, open the eiger2cbf_GUI.py shortcut

  • Fill in the boxes with information about your frames. If you are going to import more than one wedge of data into CAP, you will need to add the wedge number to the end of your output file prefix. Eg, for wedge 1, name the output “output_1” and for wedge 2, name it “output_2”. This means that CAP will read them in as runs 1 and 2 automatically.

  • Click start. The frames will start converting. When an info box pops up saying “Program completed successfully!”, your frames are ready to open with CrysalisPro. If you are doing more than one wedge/run, you will need to convert each one individually. You can click “Edit” to select the master file for the next run and give it the subsequent wedge name in the output prefix field.

 

Optional:

If you want to run eiger2cbf manually so you can select which frames to convert, eiger2cbf is in the PATH, so open a terminal and type eiger2cbf. See eiger2cbf GitHub for instructions.

 

2. Convert h5 files to cbf (at home)

Things you will need:

Option 1: Via terminal

In the folder for a collection/dataset:

  • Open a terminal

    python eiger2cbf_spawn.py [name of master h5 file] [number of instances to spawn] [filename stem for CBF]

  • Unless eiger2cbf_spawn.py is in the same folder, you will need to include the path to it in the command. 

  • In general, use 17 instances to spawn. If you add test to the end of the above command, it will tell you the fastest number of instances to spawn. 

  • If you are going to import more than one wedge of data into CAP, you will need to add the wedge number to the end of the filename stem for CBF. Eg, for wedge 1, enter “output_1” and for wedge 2, name it “output_2”. This means that CAP will read them in as runs 1 and 2 automatically.

  • For example:

    python "C:\Users\Documents\eiger2cbf\eiger2cbf_spawn.py" Thr_1_7s_Om180_K0_P0_0018_master.h5 17 Thr_1_7s_1

Option 2: Via Graphical User Interface (GUI)

You will also need a script which launches a GUI.

For this GUI to work, you will also need:

pip install gooey

  • The eiger2cbf_spawn.py script saved to your C drive (or you can edit the path to eiger2cbf_spawn.py in the eiger2cbf_GUI.py directly)

To run the GUI, go to the folder where eiger2cbf_GUI.py is saved, open a terminal there and run:

python eiger2cbf_GUI.py

This should pop up with a GUI. If it doesn’t, there is an error with the script or how it is set up. See instructions for running the script manually above.

CrysalisPro data processing (at Home and Synchrotron)

3. Importing data

  • Open an offline copy of CAP (use any old dataset to get it open). There is one saved called run1 in Documents on the Windows computer on MX1.

  • Open the import icon on the left

  • Use the known image format

  • Choose Dectris. A box will pop up to import your frames. Browse to the folder with your cbf files and select the first frame of the first run. It should automatically find the last file. eg, below the first frame ended in 1000001. cbf because it was the first frame of run 1, and the last frame ends in 2000700.cbf because it is the 700th frame of Run 2. When you are happy with this, click Save run file.

  • The program will then popup a box showing the parameters that it has found in the master file, which can be changed here if they are incorrect (they are correct in this example).

  • The program will then generate frame aliases, and then popup a window so the newly created dataset can be opened in CAP

  • Click “Open selected”, or double click the line.

  • You will then get an alert to set basic settings for the experiment:

  • Click OK, then on the next window, click “Edit options”:

  • It is usually sensible to turn on “Smart background during reflection integration” and “Single wavelength data red”

 

 

4. Indexing

If there are strong rings from the gaskets, they can be masked. 

  • Right click on the diffraction image and select "Skip d-value overlay", and “Line select (for skip d-value)”

  • Then left click and drag over a section of the gasket ring, and right click and select "skip d value". A ring should show up that overlays the gasket ring. This will act as a mask and ignore that diffraction. 

If the DAC is not facing the beam directly as Omega = 0, you need to change the value of Omega in Instrument settings to the angle at which the DAC is directly facing the beam. 

Peak hunting as normal

Unit cell finding as normal

Inspect Ewald sphere to see if cell looks sensible. If you know a cell which it should be and it has not been found, click Lattice and enter the cell. 

 

5. Data reduction with options

All as normal except for:

Step 3:

Edit special parameters

select: HP opening angle (40), reject reflection with bad profile

ALT-E, select: Mosaicity from 2D profile

Step 4:

Smart background, range = 3

Step 5:

Use friedel mates as equivalent, unless it is chiral. 

Step 6:

Space group determination, select Manual. And enter your chemical formula.

Wait for awhile... until space group determination box pops up. 

 

6. Space group determination

(Basically the same thing as going through xprep)

In E2-1 tab, check the E*E-1 values. If there are diamond peaks overlapping with sample peaks, the values will be much higher. 

7. Data reduction - Inspect data reduction results

Look at what resolution the rint gets too high (~0.44). This will be where to cut the data. 

Refinalise

Under Filters and limits, select Manual. Enter the resolution limit chosen above. 

Select Export options, and select Export sadabs .raw

Change the name of the output file if wish. 

7a. Merging

CrysalisPro

Different runs can only be processed together in CrysalisPro if the orientation matrix is the same. Aka, if the goniometer wasn't moved between collections. So you can process Wedge 1 and 2 together, or Wedge 3 and 4, or Wedge 5 and 6. This means that if Kappa or Phi were moved, and you had to recenter the crystal, you can't merge in CAP (investigation ongoing to see if we can get around this). One way of getting around this is to user Profit merge in the Data reduction tab. This can only be done after each pair of datasets has been processed in CAP, and it does not scale the datasets together. Further work ongoing to see if we can scale and merge all the datasets together.

To merge with CAP:

  1. Open any one of the experiments in CAP

  2. Go to the Data reduction tab and click the small arrow button on the ‘Data Reduction’ tab and select
    ‘Proffit merge’ to open the interface for merging data sets.

  3. Use the ‘Add’ button ’ to add reflection files. If a reflection file is in a different directory, browse to that directory and select the desired file. Note that only reflection files with extension ‘.rrprof’ can be used. The main panel of the interface shows the reflection files added and an estimate of the completeness of the
    merged data. If a file needs to be removed from the list, select it and click on ‘Delete’.

  4. Edit the name of the output file i.e., the merged file, in the panel at the very bottom of the
    interface and click ‘OK’.

  5. Open the Refinalizer. From the pull-down menu at the bottom of the interface, select the
    merged reflection fill created in the previous step, as shown by the arrow below.

  1. Finish processing as normal with this merged hkl.

sadabs

Read in multiple raw files into sadabs and process as normal. 

xprep

Put hkls together in a new folder and merge as normal. 

8. Solving data

Use ambient model

Get the ins file from the ambient dataset (if space group and unit cell the same). Or a previous solution with the same unit cell and space group. 

Copy from the SFAC command to the end. 

In the new ins file for the HP dataset, delete everything from SFAC down and replace with ambient ins file. 

Open in Olex and refine as normal. Usually a good idea to remove Hydrogens and make unisotropic before first refinement. 

 

9. Refinement

Check Fobs/Fcalc. Should be a straight line. 

If it is flattening out at the top it means it was overexposed. Trying applying extinction to address this.

Check the Normal Probability graph. Reflections not nicely on the line can be omitted. Check where they were in the Fobs vs Fcalc, and see if the Fobs vs Fcalc is closer to being a straight line.