Fourier transforms in cryo-EM | Convolution, Gaussian filters, Fourier Shell Correlation, FFT
Summary
TLDRPavel Afanasyev's lecture on Fourier transforms in cryo-EM covers the importance of these mathematical tools in analyzing and reconstructing data from electron microscopy. The lecture begins with the basics of periodic functions and Fourier transforms, explaining how complex functions can be decomposed into simpler sinusoidal components. It highlights the historical context of Fourier transforms in electron microscopy, their applications in data collection, and the significance of sampling and the Nyquist frequency for achieving high resolution. The lecture also discusses practical applications such as correlation, convolution, and filtering techniques, emphasizing the convenience of using Fourier transforms for data analysis. Afanasyev encourages the audience to utilize Fourier transforms as diagnostic tools in their work and recommends further resources for learning.
Takeaways
- 📊 Fourier transforms are essential in cryo-EM for data analysis.
- 🔍 Understanding periodic functions is key to grasping Fourier transforms.
- 📈 The Nyquist frequency determines the maximum resolution achievable.
- 🖼️ Fourier transforms allow decomposition of complex functions into simpler components.
- 🔄 The Fast Fourier Transform (FFT) is an efficient algorithm for calculations.
- 🧩 Convolution and correlation are important operations in image analysis.
- 🛠️ Filtering techniques (low-pass, high-pass) help in enhancing image quality.
- 📚 Recommended resources include Stanford lectures and classic Fourier analysis books.
- 🔬 The Central Slice Theorem aids in reconstructing 3D objects from 2D data.
- 🧪 Fourier transforms serve as diagnostic tools in data processing.
Timeline
- 00:00:00 - 00:05:00
Pavel Afanasyev introduces his lecture on the use of Fourier transforms in cryo-electron microscopy (cryo-EM), emphasizing the importance of understanding Fourier transforms for data analysis in this field. He outlines the structure of the lecture, which will cover one-dimensional periodic functions, two-dimensional images, and applications in electron microscopy.
- 00:05:00 - 00:10:00
The historical context of Fourier transforms in electron microscopy is discussed, highlighting the contributions of De Rosier and Klug in the mid-20th century. Their work on bacteriophage T4 tail reconstruction using Fourier analysis is presented as a foundational example of Fourier transforms' significance in cryo-EM.
- 00:10:00 - 00:15:00
The lecture begins with the definition of periodic functions, explaining key parameters such as amplitude, period, and phase. The relationship between frequency and period is introduced, emphasizing the relevance of these concepts in cryo-EM data analysis.
- 00:15:00 - 00:20:00
The concept of summing periodic functions is introduced, leading to the idea of decomposition, where any periodic function can be represented as a sum of sinusoids. The Fourier transform is defined as a mathematical tool that allows this decomposition, with a focus on its practical applications in cryo-EM.
- 00:20:00 - 00:25:00
The properties of Fourier transforms are discussed, including the ability to invert the transform, allowing for the reconstruction of the original function from its Fourier components. The importance of sampling and the Nyquist frequency in relation to resolution in cryo-EM is emphasized.
- 00:25:00 - 00:30:00
The distinction between real space and Fourier space is made, with an explanation of how functions can be decomposed into their frequency components. The concept of amplitude spectrum is introduced, highlighting the symmetry of Fourier transforms and the significance of higher frequency terms in data analysis.
- 00:30:00 - 00:35:00
The lecture transitions to two-dimensional Fourier transforms, using images as examples. The importance of pixel intensity and dimensionality in image representation is discussed, along with the process of decomposing images into their Fourier components.
- 00:35:00 - 00:40:00
The concept of scaling in both real and Fourier space is introduced, explaining how images can be binned and how this affects pixel size and resolution. The process of Fourier cropping is also explained as a method for efficient image processing.
- 00:40:00 - 00:45:00
Fourier filtering techniques are introduced, including low-pass, high-pass, and band-pass filters. The use of Gaussian functions for filtering in cryo-EM is emphasized, along with the importance of avoiding sharp edges in masks to prevent artifacts in the inverse Fourier transform.
- 00:45:00 - 00:59:44
The lecture concludes with a discussion on the application of Fourier transforms in 3D reconstruction and convolution, highlighting the efficiency of using Fourier transforms for operations like deconvolution and correlation in cryo-EM data analysis. The significance of Fourier Shell Correlation in determining resolution is also explained, along with recommendations for further reading on the topic.
Mind Map
Video Q&A
What is the main topic of the lecture?
The lecture focuses on the use of Fourier transforms in cryo-electron microscopy (cryo-EM).
Who is the lecturer?
The lecturer is Pavel Afanasyev.
What are the key components of Fourier transforms discussed?
The key components include amplitude, phase, and frequency.
What is the significance of the Nyquist frequency?
The Nyquist frequency is crucial for determining the maximum theoretical resolution achievable in cryo-EM.
What are some applications of Fourier transforms in cryo-EM?
Applications include data collection, CTF correction, validation of 3D reconstructions, and filtering techniques.
What is the difference between real space and Fourier space?
Real space refers to the original data domain, while Fourier space is the domain after applying Fourier transforms.
What is the Fast Fourier Transform (FFT)?
The FFT is an efficient algorithm for computing Fourier transforms.
What is the Central Slice Theorem?
The Central Slice Theorem allows for the reconstruction of 3D objects from their 2D projections in Fourier space.
What are low-pass and high-pass filters?
Low-pass filters allow low-frequency information to pass, while high-pass filters allow high-frequency information to pass.
What resources are recommended for further learning?
Recommended resources include lectures from Stanford University and classic books on Fourier analysis.
View more video summaries
These Jobs Won't Exist In 24 Months | Daniel Priestley, Bret Weinstein, Amjad Masad | Diary Of A CEO
POLITICAL THEORY – Jean-Jacques Rousseau
Hedonism: The Pursuit of Happiness
Future of Logistics
HOME by Yann Arthus Bertrand - Full Movie (English version) - Documentary
English Pronunciation | The Letter 'A' | 9 Ways to Pronounce the Letter A
- 00:00:00Hello my name is Pavel Afanasyev and first I would like to apologize I couldn't make it to Brazil.
- 00:00:06Nevertheless, I recorded the lecture for you and I hope you'll find it useful.
- 00:00:16The title of the lecture is "the use of Fourier transforms in cryo-EM".
- 00:00:20It's very difficult to cover the full topic of Fourier transforms within one hour.
- 00:00:25Therefore, the idea of this lecture is to illustrate to you why Fourier
- 00:00:32transforms are so important in cryo-EM and to give you a background for an
- 00:00:39intuitive use and for intuitive understanding of the maths behind.
- 00:00:48So, my talk will consist of several sections I will start with one dimensional periodic
- 00:00:56functions, I'll cover the idea of the Fourier transforms of those.
- 00:01:01Then, I'll switch to two-dimensional periodic functions (or images).
- 00:01:06We will talk about Fourier transforms of the images and we will also talk a little
- 00:01:12bit about the Fourier transforms of the 3D functions or volumes (cryo-EM maps).
- 00:01:20And finally, I'll try to discuss the applications of where it transforms in electron microscopy.
- 00:01:29The first time Fourier transforms were used in electron microscopy was in the
- 00:01:37middle of the 20th century, and that happened thanks to the crystallographers
- 00:01:43who were starting doing electron microscopy. De Rosier and Klug were two scientists who studied
- 00:01:51the tail of bacteriophage T4 and they used methods from X-ray crystallography for determination of
- 00:02:023D-reconstruction from an object, and they used these methods of the Fourier analysis in order to
- 00:02:11obtain the structure of the bacteriophage T4 tail. So, in this lecture I will try to explain what
- 00:02:21exactly they were doing and what are the methods behind.
- 00:02:26Today, it's very difficult to imagine the field of cryoelectron microscopy without Fourier
- 00:02:32transforms. And pretty much starting with the data collection, we are using Fourier transforms
- 00:02:39everywhere: from the CTF-correction (it's pre-processing stages of the data analysis) up to
- 00:02:47the validation of your final 3D-reconstructions using Fourier Shell Correlation.
- 00:02:55Let's start with the very basics and first I would like to introduce (or define) periodic functions.
- 00:03:04So, what is periodic? Periodic means the function, where you can add any
- 00:03:10period to the to the function and then you'll get the same function back.
- 00:03:14The easiest way to imagine that would be if you look at the simple sinusoid.
- 00:03:23This is the function sin(x) and the the function sin(x) can be generalized in these terms: where
- 00:03:33"A" would be an amplitude and we know that A=1 in this case, so the amplitude of a function would be
- 00:03:44the distance between the x-axis and the highest crest of the function.
- 00:03:55In a way this is how high the wave of the function is.
- 00:04:00Now, the second important parameter would be the period.
- 00:04:06And the period would be the distance between two crests of the function.
- 00:04:13So, in case of our function... (our sinusoid function) the period would be equal to 2 pi.
- 00:04:24Also period (you'll find sometimes), is known as a wavelength of the function.
- 00:04:35Now, the third parameter which we'll be talking about is the phase.
- 00:04:40And to talk about the phase (if we want to talk about the phase of the function),
- 00:04:48it's easier if you imagine another function, for example sin(x - pi/4).
- 00:04:55And pi/4 would be the phase, and that would be the distance between these two functions.
- 00:05:03So, this is how... (if you imagine that this axis would be time) it means that how
- 00:05:10slow or what is the delay of these two functions. And it's also sometimes denoted as a phase shift.
- 00:05:22So, knowing these two functions and looking at this general equation we can easily assign
- 00:05:29values for the phase, for the period or the wavelength, and for the amplitude.
- 00:05:35Now, in cryo-EM and for us it would be important also to talk about the frequencies,
- 00:05:41and frequency is related to the period (or the wavelength) as 1/T.
- 00:05:49So, the period, sorry the frequency, would illustrate how fast the function will be changing,
- 00:05:59and it's defined as a number of cycles per second or x axis unit.
- 00:06:09In cryo-EM, as I mentioned, we would be focused in definitions of any functions using these three
- 00:06:17variables: amplitude, phase and the frequency. So, the simplest operation we could think about,
- 00:06:25when we work with periodic functions, would be summing.
- 00:06:29So, what means to sum or to average two functions? Again, let's consider one function - the first one
- 00:06:37which we talked about already, it would be sin(x). And the second one will have a different amplitude
- 00:06:46and a different frequency - it will be 1/3*sin(3x).
- 00:06:50So, if we sum these two functions you'll see that the second function has the minimum where
- 00:06:57the first one has the maximum, and if we average or if we sum these two functions,
- 00:07:01we'll get something like this, so this will be the sum of these two functions.
- 00:07:06Let's consider the reverse operation which is called decomposition.
- 00:07:11So, the idea of the Fourier transforms actually is in the following: any periodic function can
- 00:07:18be represented as a sum of sinusoids or as the sum of other periodic functions.
- 00:07:25For example, let's consider a more complicated function which would look like this,
- 00:07:31and turns out that this function can be represented as a sum of the following functions,
- 00:07:39which you can see here on the right. So, the first one, it would be the sin(x),
- 00:07:44which you can see here, and then, if we add to sin(x) a second function
- 00:07:48which would look like this, the sum of these two would be here.
- 00:07:53The blue curve will be the sum of these two functions.
- 00:07:58Now, if we consider the third function, which will have higher frequency,
- 00:08:05and if we average these three functions, they will already
- 00:08:12approximate the original function quite well. But moreover, if we take the fourth function
- 00:08:19we can fully represent or we can fully reconstruct the original functions which we started with,
- 00:08:27so the four the sum of these four functions would
- 00:08:31be the original function we wanted to decompose it into.
- 00:08:35And this is called decomposition - representation of the function as a sum of other functions.
- 00:08:42And you might ask: "How do we get all these values of the amplitudes and the phases?
- 00:08:50How do we know how to represent or how to decompose the function into?"
- 00:08:55And that's exactly what the Fourier transform would be doing.
- 00:08:59So, the Fourier transform allows you to decompose any periodic function
- 00:09:03into the series of sine waves. And the person who came up with
- 00:09:09such an idea and who described it was the French mathematician Fourier.
- 00:09:14He came up with this formula. This formula might look a bit complicated for you, but actually it's
- 00:09:20very simple: so these two terms, they are very much related because sine and cosine function
- 00:09:25are different just by addition of of pi/2. So, we can just focus on one of them,
- 00:09:33right? And then, we also have a constant here which you can ignore for now.
- 00:09:38So basically, this function would be the sum of these sinusoids. And in order to get this
- 00:09:47amplitude which, Bm would actually be, we would have to calculate these integrals.
- 00:09:58A bit complicated, but there are ways how to calculate those,
- 00:10:02and A and B would be the amplitude here. So, there is an equation which allows to
- 00:10:09perform such operations of the composition of functions as a sum of some other functions.
- 00:10:18Now, the most important part about that is that the Fourier transform can be fully inverted.
- 00:10:24What does that mean? If we have our function and we call it the function which is in "real space",
- 00:10:32and if we perform the Fourier transform (if we decompose it as a sum of some other functions),
- 00:10:38we'll be working now in so-called "reciprocal" or "Fourier space".
- 00:10:43So, we perform one Fourier transform and we get
- 00:10:48another function which is a sum of some functions, and then, if we perform an
- 00:10:54inverse Fourier transform, we would be able to get our function back in real space.
- 00:11:03Now, as I already mentioned, such kind of decomposition
- 00:11:09would imply that the terms which we would be using (or the functions which would be
- 00:11:16using) they would have certain properties. And you can already see here some trends.
- 00:11:24So, as we described before, we can calculate the amplitude period and phase for each of these
- 00:11:34functions and we will see that the, for example, the wavelength of the first function would be 2 pi
- 00:11:42and the frequency would would be 1/2 pi. And then, we can again determine the
- 00:11:48wavelengths for the second one and the frequency for the second one.
- 00:11:51And what we can see here is that the frequency for each further function, for each next term in this
- 00:12:03decomposition will be actually higher and higher. We can see that the wavelength is getting smaller
- 00:12:08and the frequency is getting higher. So, the highest frequency which we'll
- 00:12:15be using for our decomposition will be called the "Nyquist frequency".
- 00:12:20And deciding on how... what would be the size of these frequencies or would be the size of the
- 00:12:34very highest frequency for the decomposition is related to the term which is called "sampling".
- 00:12:41So, it's in case of 1D case (a one dimensional case) it would be just
- 00:12:46defined by the smallest wavelength, which we would be using for the decomposition.
- 00:12:50And in case of a two-dimensional (for two-dimensional case for the images), or for the
- 00:12:58three-dimensional case that would be determined by the pixel size or by the voxel size,
- 00:13:04and in turn, these are defined just by your magnification of the microscope.
- 00:13:10Why is it important? It's important because of the final resolution we would be aiming
- 00:13:21for and and it's important to be able to decompose any function as fine as possible.
- 00:13:31Why? Because if we use frequencies which would be too large, we would not be able to
- 00:13:39reconstruct the original signal and that's what the Nyquist-Shannon sampling theorem is about.
- 00:13:45It says: "signal sample at rate f can be fully reconstructed,
- 00:13:49if it contains only frequency components below half sampling frequency.
- 00:13:56So, that means that in order to fully reconstruct your signal
- 00:14:01you need to have certain type of sampling or you need to have certain type of your pixel size.
- 00:14:08And in case of cryoelectron microscopy, where the Nyquist frequency term will be
- 00:14:17used, it is in terms of the maximum theoretical resolution which you would be able to achieve.
- 00:14:24And that would be equal to one over two times your pixel size.
- 00:14:28So now, when we defined the Fourier transforms, when we defined the sampling, it's very important
- 00:14:39again to denote two domains which we are working with. The so-called "real space" domain or
- 00:14:47"spatial domain" and "Fourier-space domain" or we will know that it's called a "frequency domain".
- 00:14:54So, we were able to decompose this function as a sum of these four
- 00:15:02and for each of these four we know its frequencies and we know the amplitudes.
- 00:15:09And we can create a plot of the frequencies which would be the frequencies and amplitudes.
- 00:15:19And Fourier transforms have properties of being symmetric, and that's why we
- 00:15:26have four amplitudes corresponding to its frequencies as positive, and four as negative.
- 00:15:33So in principle, we can, if we know this side of the plot,
- 00:15:38we would also know this side of the plot. And this is called "amplitude spectrum".
- 00:15:43So any function would be... it's possible, would be possible,
- 00:15:48to think about it in terms of their amplitudes phases and in terms of their frequencies.
- 00:15:54Now, as I said before, it's very important to have higher frequency terms in the decomposition
- 00:16:02and these higher frequency terms would be defining how good we decompose the function.
- 00:16:11And, for example, if you have some kind of an exotic function, which would look like this,
- 00:16:16we would be taking more sinusoid functions with higher and higher frequencies.
- 00:16:23And they will be describing this function better and better until
- 00:16:28it converges and I have an example here and I hope I can start it this video...
- 00:16:39So, you can see here that with adding more and more terms, the function is
- 00:16:46being approximated better and better. And as I noticed, as I said before,
- 00:16:53the frequency of the very last term the highest term will be called Nyquist frequency, and it
- 00:16:58reflects the sampling which we have chosen. You can already see here that for some sharp
- 00:17:10features we would need to have a lot of different functions.
- 00:17:16And the truth is that actually for the very sharp-like edges, would need
- 00:17:23an infinite number of functions in order to decompose the original function completely.
- 00:17:33So, finer sampling implies using more functions with higher frequencies.
- 00:17:39And if you are familiar with complex numbers it's very important to note that
- 00:17:46the Fourier transforms can be expressed in a slightly different form,
- 00:17:52and in terms of the complex numbers we know that the complex numbers would have their amplitudes
- 00:17:59and faces just the numbers can be expressed slightly different, and there's just another
- 00:18:03representation of Fourier transform and sometimes you'll be seeing that Fourier transforms will be
- 00:18:08expressed in terms of their amplitudes... And sometimes Fourier transforms would be
- 00:18:15expressed in this form - in the form of complex numbers with their amplitudes and their phases.
- 00:18:24So, I already um mentioned to you that there are issues with the
- 00:18:30sharp features of the functions, and that would need an infinite number of functions.
- 00:18:35And how... and you might be asking, how do we deal with that in real life?
- 00:18:41Because instead of a sum of these functions we will have to
- 00:18:46integrate over the infinite number of terms. And you can easily, or you can not easily,
- 00:18:53but you can imagine that actually the images, which you will be working with,
- 00:18:59they are consisting of pixels and, therefore, this decomposition of image into a pixel would
- 00:19:11be somewhat its own periodic function. And when we will be talking about Fourier
- 00:19:18transforms of the images we would be talking about so-called "Discrete Fourier Transform",
- 00:19:24which is implemented using "Fast Forward Transform" algorithm, and sometimes in some
- 00:19:30image processing programs you will see this term "FFT", and that means it's just an algorithm for
- 00:19:38implementation of the Fourier transform. And it's very fast - using this algorithm
- 00:19:42you are able to calculate the Fourier transforms very efficiently.
- 00:19:49And these Fourier transforms would be expressed or these coefficients, the Fourier components
- 00:19:58would be expressed as a sum rather than as an integration over the infinite number of terms.
- 00:20:09So, it might have been a bit complicated for you, and you might ask why would we bother, why
- 00:20:15would we need to represent it in terms of... why would we need to decompose any kind of function?
- 00:20:21And the answer for that is because it is very convenient, and later on you'll see (there is
- 00:20:28a spoiler) that you will be able to compute it much faster using Fourier transform, compute
- 00:20:37certain things much faster than if you would have to compute them in real space.
- 00:20:44So let's now generalize our approach for the 2D case and let's talk about the images.
- 00:20:51This is an image of 20S proteasome dataset, it's a micrograph. This was collected on the
- 00:21:00Falcon 3 camera which has 4096 pixels in this dimension and 4096 pixels in this dimension.
- 00:21:08Nowadays, you might be using already the next generation of the detector, for example Falcon
- 00:21:134 detector, but all these detectors Falcon three two and one and four have the same dimensions:
- 00:21:204K x 4K with a size of 16.7 megapixels. Some of you might be using different cameras,
- 00:21:28for example, Gatan K2 or Gatan K3, and they have slightly different dimensions but it's
- 00:21:36comparable in terms of the magnitude they are (the size of these detectors).
- 00:21:42Moreover, if you talk about just digital cameras the camera in the latest iPhone
- 00:21:51will be even bigger than these cameras (which
- 00:21:55are used in cryo-EM) and it would be composed of close to 50 megapixels.
- 00:22:03Now let's talk just about.. about photos for simplicity.
- 00:22:08And let's consider such a photo of Marin van Heel,
- 00:22:12and he's holding here a camera. And if we start zooming in,
- 00:22:20in this photograph, we will see the camera zoomed in and then we'll see this lens, and if we zoom
- 00:22:29in further, we will start seeing actually an image with some pixels. It's a pixelated image,
- 00:22:35and in this case it will be 20 by 20. So, if we zoom in at any other area of
- 00:22:41these photographs of this photograph, we will see this pixelated zoomed-in image. What does it mean?
- 00:22:49So again, here I have 20 pixels in this dimension and 20 pixels in this dimension (this is the
- 00:22:56cropped area which we're talking about), like in case of the cameras we were talking about.
- 00:23:03So, each pixel will have its own intensity: it might be whitish or it might be more
- 00:23:12grayish or it might be very dark. So, that means each pixel will have its
- 00:23:18own intensity, and depending on how white or how black it would be, we can actually
- 00:23:28assign different values to these intensities. And if we consider such a pellet, we can say that
- 00:23:38very dark areas would be having intensity of zero, and very white would be having intensity of 255.
- 00:23:49Why 255? Or 255 plus 0 will be 256, this is 2 to the power of 8. And that describes a
- 00:24:008-bit imaging, so that means that we'll have 256 shades of grey in our image.
- 00:24:08If we consider 16-bit, we will have much larger number of shades of gray.
- 00:24:15So now, if we represent each pixel of the image in terms of its intensities,
- 00:24:24we can assign a number to each pixel, which would be the number of them,
- 00:24:31or which would be the intensity value for each pixel.
- 00:24:36And then, as you can see here, very bright areas would have higher values, for example this white
- 00:24:44one (whitish) would be 183, and then very dark would be having something like 18 or 19 and so on.
- 00:24:54And we're talking about dimensionality all the time and then before we talked about 1D curves
- 00:25:02one-dimensional curves, and we were talking about sinusoids and so on...
- 00:25:06But what 1D curve actually means, is that we have only a single coordinate and for this coordinate
- 00:25:15we can have a pair of the value which would be another value in this plot,
- 00:25:22and this is the coordinate value.
- 00:25:24So, we are always plotting our curves
- 00:25:29in this grid, but what we actually can do, we can assign the intensity value for this curve
- 00:25:41and, for example, everything which will have higher intensity depending on the x-coordinate
- 00:25:48will be having... will be depicted as white here. So... and this is 1D curve.
- 00:25:58So, in case of the images we will talk again in two dimensions, but now in terms
- 00:26:05of the X and Y coordinates. So, each pixel will have
- 00:26:09its intensity, but also each pixel will have coordinates X and Y.
- 00:26:13And in case of 1D we have only one coordinate. In case of 3D we'll have
- 00:26:18three coordinates plus the intensities. This is about the dimensionality.
- 00:26:24So now, as we noted before, since the image can be represented in terms of
- 00:26:32its intensities and we're going now in the world of 2D, we can represent an image as
- 00:26:40a linear combination of some other images. And in this case, if you want to decompose
- 00:26:48an image, which would be 3x3 - a very simple image with its own intensities
- 00:26:54we can decompose it in real space. We can just decompose it as a sum
- 00:27:00of such so-called "base vectors" with a certain coefficients which would be
- 00:27:09similar to the amplitudes.
- 00:27:11So, any image in real space can be decomposed in terms of again the
- 00:27:15sum of some other images and this is a good analogy to think about Fourier transforms.
- 00:27:26And let's talk about the actual Fourier transforms and what the Fourier transforms..
- 00:27:34how they would look like, if we perform Fourier transform of very simple functions.
- 00:27:41For example, let's perform a Fourier transform of this sinusoid but in 2D. So,
- 00:27:48we have oscillations only in one direction here. So, the Fourier transform of such an image
- 00:27:56would look like two dots in this direction. So, why two dots? It's because... two because,
- 00:28:05as I mentioned before, Fourier transforms have the symmetry,
- 00:28:12and um it has it will be exactly the same in the negative direction as in the positive direction.
- 00:28:19Now, if we imagine another Fourier transform of this... of such an image, which is similar but
- 00:28:25rotated by 90 degrees, we will see two dots, which would be again rotated by 90 degrees.
- 00:28:32What's more interesting is to consider another sinusoid which
- 00:28:36would be oscillating but at higher frequencies.
- 00:28:41So, the Fourier transform in Fourier space (this image) would look like this.
- 00:28:46And now comparing these two, we can see that the frequency here is higher, and that means that the
- 00:28:53distance between these two dots would be larger. Everything which is closer to zero here to the
- 00:29:00centre of the image would be representing lower frequency information, everything
- 00:29:06which would be going towards the edge, will be representing the higher frequency information.
- 00:29:12And again, you can imagine easily, that if we have something like this,
- 00:29:16again this operation is equivalent to the rotation.
- 00:29:20In the perpendicular direction to the distribution of the wave of these
- 00:29:26waves we don't have any oscillations, yes? That's why in this direction we have zero.
- 00:29:35And if we have a superposition of these and the perpendicular function which would look
- 00:29:40like these checkers, obviously, will have something like four dots here.
- 00:29:47So, using such decomposition we can talk about representing our real space image from before
- 00:29:57(and that represents probably number "1" in the lens).
- 00:30:01It's in real space, we can decompose the image in 2D in real space as a sum of these of these images
- 00:30:15from the Fourier space. So, the same way how we were talking about 1D Fourier transform,
- 00:30:21we can create a 2D Fourier transform and to decompose any image as a linear combination
- 00:30:30of the Fourier components.
- 00:30:35And the good thing about it is that the same way as we are talking about the 1D
- 00:30:43Fourier transform, you can create Fourier transforms as many times as you want.
- 00:30:48You can transform there and back, you can create the forward in an inverse Fourier
- 00:30:53transform and work with. It it's very convenient.
- 00:30:56OK, the same way as we were talking about the amplitude spectrum in 1D case, we can talk about
- 00:31:03the amplitude spectrum in 2D case. So, if we take a test image,
- 00:31:08if we create a Fourier transform we will get this amplitude spectrum.
- 00:31:13And when we're talking.. when we're in the world of cryo-EM, we will be
- 00:31:17talking about so-called "power spectrum". Power spectrum and amplitude spectrum basically
- 00:31:21is the same thing, with the difference that amplitude squared would be the power.
- 00:31:30And it's just easier to talk in terms of the power spectrum because it would be reflecting
- 00:31:36the actual intensities which we see in the images.
- 00:31:39And together with the amplitude spectrum you should also realize that we always
- 00:31:43have, when we perform a Fourier transform, we'll always have phases.
- 00:31:50And this would be represented by so-called "phase spectrum". So when
- 00:31:54we create the Fourier transform, we have amplitudes and phases at the same time.
- 00:31:59Now, as I said, as I briefly mentioned before, everything which is closer to the
- 00:32:05centre of this image, would correspond to the lower frequency information
- 00:32:10and everything which would be going farther from the centre up to here, will be corresponding
- 00:32:17to higher frequency information with the highest frequency being a Nyquist frequency.
- 00:32:24And let's consider a very simple operation ,which we can do in both spaces:
- 00:32:30real space and Fourier space. Let's talk about scaling.
- 00:32:35So, if we have an image, which would be 6x6 pixels, and as I mentioned above,
- 00:32:41each pixel will have its own intensity, what we can do - we can scale our image.
- 00:32:48For example, we want to bin our image by the factor of two.
- 00:32:52What would it mean? It means that if we have an image of 6x6 pixels,
- 00:32:58if we bin by the factor of two, we will get an image of 3x3. If we had sampling (or pixel size)
- 00:33:05of 1 Angstrom/pixel here, then the sampling here would be 2 Angstrom/pixel after being,
- 00:33:15and each pixel on the binned image will be an average of four pixels from this image. S
- 00:33:26o, an average of 0, 0, 1 and 3 will be 1. Now, we can move further and then we can
- 00:33:32take an average of these four which will be four and an average of these four, which will be two.
- 00:33:38And this way we can calculate the values for all the other pixels in this image which will
- 00:33:48not necessarily be integers and then this way we would be performing the beginning in real space.
- 00:33:57And as I said, it's important to know that the sampling (or the pixel size)
- 00:34:00of the binned image will be twice larger than the pixel size of the original one.
- 00:34:07How would we do the scaling in Fourier space? In Fourier space we can take an image
- 00:34:13and we can create a Fourier transform. And if you create a Fourier transform of
- 00:34:17an image of a micrograph in cryo-EM, you'll see this pattern which is called the "Thon rings",
- 00:34:25and in later lectures you will hear about the Thon rings.
- 00:34:29So, what we can do in order to bin this image of 4K by 4K?
- 00:34:35We can take the central part of the image, and the central part will be twice smaller than in these
- 00:34:45dimensions: it will be 2048 pixels by 2048. And if here, as I said, will be the Nyquist,
- 00:34:56the original Nyquist frequency, here would be the Nyquist frequency over 2.
- 00:35:02And then, what we can do, we can just crop it. We can crop the original image
- 00:35:09and create an inverse Fourier transform. And if we create an inverse Fourier transform,
- 00:35:14our original image will be binned to 2k x 2k, and the pixel size will be twice coarser
- 00:35:22than the original pixel size. Using this operation, we can define the cropping area,
- 00:35:31which would be pretty much any, and that means that we can bin also [by] non-integer,
- 00:35:37and this type of binning allows you not to introduce any kind of interpolations, which
- 00:35:45is much better than doing binning in real space. And this operation is called "Fourier cropping"
- 00:35:52so we use Fourier transforms for binning, for example.
- 00:35:57Now, another operation which you can be doing using Fourier transforms,
- 00:36:02would be Fourier filtering. Again we take an a test image, and this image is very nice actually,
- 00:36:08because it contains lots of high-frequency information here in the clothes of the lady,
- 00:36:12and also these checkers on the tablecloth. So, [for] any image, as I said, we can
- 00:36:20perform the Fourier transform of this image, and then we will get an amplitude spectrum.
- 00:36:26So, again, what we can do with this amplitude spectrum,
- 00:36:31we can consider a distribution of different amplitudes.
- 00:36:36And what if we focus now only on the middle part of this amplitude spectrum,
- 00:36:43what if we set everything else to zero? So, we'll have only the central part of
- 00:36:49this amplitude spectrum, and then, if we create an inverse Fourier transform,
- 00:36:54we'll get an image like this. So, all the high-frequency information
- 00:36:58all the tiny details in the image will be gone, and this is called "low-pass filter".
- 00:37:05We are allowing to pass only low-frequency information that's why it's called low-pass.
- 00:37:12Now, if we consider another case, where we cut off everything - like 20 percent from the original
- 00:37:25data (in terms of the the frequency domain), if we cut off everything up to Nyquist over five
- 00:37:34in the low-frequency information, and we create an inverse Fourier transform,
- 00:37:38we will get an image like this. And we see that all the high-frequency information
- 00:37:43is preserved, however all the shades, all the features, like large features (which correspond to
- 00:37:54all the features in the image or the gradient of Illumination, stuff like that) will be gone,
- 00:38:02and using such an operation, allows us to focus on the high-frequency details:
- 00:38:10we are passing only high frequencies, and that's why this operation is called "high-pass filter".
- 00:38:17We can also consider a combination of both, and we can use a ring over the Fourier transform over the
- 00:38:24amplitude spectrum and then that will be called a "band-pass filter" - so we are allowing to pass
- 00:38:35a certain fraction of frequencies and the high frequencies will be
- 00:38:39excluded here as well as some low frequencies will also be excluded.
- 00:38:48And in cryo-EM for applying this low-pass,
- 00:38:53band-pass and high-pass filters we will be using so-called "Gaussian" functions.
- 00:38:58Why Gaussian, and they would look like this, because the Fourier transform of the Gaussian
- 00:39:03function would be a Gaussian function. Why is it important to remember?
- 00:39:07Let's consider four images in the real space, and these four images, which would
- 00:39:17look like something very broad and in the end to also look as the delta-function pretty much.
- 00:39:24It's a dot, so a very very narrow area - so in 2D real space (in the real space),
- 00:39:35these images would look like this. But in Fourier space everything which is broad,
- 00:39:40will be condensed to a very tiny point. If it's close to infinity (the distribution
- 00:39:47of this function), that will be a delta function in Fourier space, and inversely,
- 00:39:55if there is something very broad in Fourier space, it will be very tiny in the real space.
- 00:40:04We know the relation of the Gaussian functions and Gaussian filters actually
- 00:40:10allow us to go there and back in case of the Fourier transforms.
- 00:40:17And Gaussian means that we use very smooth edges in the masks which we are applying for the images.
- 00:40:29Again, let's consider this is the Fourier transform of one image, it's an amplitude
- 00:40:34spectrum of the of the test image, and you can see here if you're applying a band-pass filter,
- 00:40:41the areas of the of the mask the edges of the mask will be very soft and that's how it should
- 00:40:49be in cryo-EM, because if edges of the mask are very sharp, and we know that the sharp edges,
- 00:40:57if it's something very small, it's a very rapid transition - it might introduce certain artefacts,
- 00:41:04if we create an inverse Fourier transform. That means, that we would we should never
- 00:41:10use sharp-edged masks and in different software packages there are parameters which correspond to
- 00:41:20the soft edges of the mask, and you should always use these masks, which would have these soft
- 00:41:27edges, because if you don't, you might introduce certain artefacts, and this is an inverse Fourier
- 00:41:33transform of an image which was filtered with a mask in Fourier space with a sharp edge.
- 00:41:41OK, so we can go further in terms of the dimensionality and we can define also the
- 00:41:49Fourier transforms of 3D objects, or if you're a mathematician, you can go up to N dimensions and
- 00:41:57you can think in terms of higher dimensions. The most important is just to realize that
- 00:42:03this generalization is possible and we will have all the all the properties of Fourier transform,
- 00:42:11which can be applied to the 3D case. We started with an example of this paper of
- 00:42:21the reconstruction of the tale of bacteriophage T4, and actually what was done in this work,
- 00:42:31what they [did], they they took a Fourier transforms of their images of the recorded images,
- 00:42:38and using the Fourier analysis they were able to figure out the spatial
- 00:42:45orientation between these images and taking advantage of the symmetry to reconstruct..
- 00:42:52to perform an inverse Fourier transform of these 2D images into 3D,
- 00:42:57using so-called "Central Slice Theorem". It's a bit.. it might sound a bit complicated
- 00:43:04"The Central Slice Theorem", but what it's actually doing, it allows to get the 3D
- 00:43:10orientation of the 2D objects in in Fourier space, using an inverse Fourier transform.
- 00:43:18In this part of the talk I will try to illustrate a few more cases,
- 00:43:25and to demonstrate what else Fourier transforms can be useful for. Let's talk about convolution.
- 00:43:34To understand the concept of convolution, you need to imagine two functions: first a function which
- 00:43:41would look like this and then a function which would be a set of impulses, for example in this
- 00:43:49case it's impulses with different amplitudes, and one is negative and there are four of them.
- 00:43:56So, what convolution would be doing is actually.. it's blending these two functions.
- 00:44:03And convolution would be expressed using the following mathematical equation.
- 00:44:10The symbol for convolution would be this encircled cross.
- 00:44:15And it might be a bit difficult to think about this in terms of the integrals, in terms of these
- 00:44:20functions and so on, but it's more important to understand what the convolution is actually
- 00:44:28doing. So, in one of them lecture notes from one of the Stanford University professors I found the
- 00:44:37following phrase, that... when he was asked and the topic of the chapter was what is convolution,
- 00:44:48and then he is noting that convolution is what convolution does. So, don't try to
- 00:44:53overthink the maths, just try to focus on their actual properties and the the actual operation.
- 00:45:03Let's consider another example, and this would be the 2D case and convolve two functions:
- 00:45:12one function would be an image or in this case it's kind of.. it's like a
- 00:45:19crystal lattice basically with ones and zeros everywhere, and the second function will be
- 00:45:25the flower. And when we're blending these two functions, we're getting something like that.
- 00:45:30The same principle and this is what would be happening when we're talking about X-ray
- 00:45:36crystallography or the same principle you'll find you'll find in the convolutional neural networks.
- 00:45:46What's important to note is again, to compute such an operation,
- 00:45:53you would be using the integral equation, and integration is very computationally heavy
- 00:46:05have a task and it takes quite a while to to calculate integrals,
- 00:46:10but a calculation of the Fourier transforms, as I said, using the FFT algorithm is very fast.
- 00:46:18What we can do, we can actually calculate the Fourier transform of.. or we can apply the Fourier
- 00:46:27transform (at least on the paper) to this part of the equation and to this part of the equation.
- 00:46:33And if we apply Fourier transform, the Fourier transform of this double integral
- 00:46:40will be the multiplication of these functions - of the Fourier transforms of these functions.
- 00:46:48So, that means that instead of integration, we can calculate a
- 00:46:54multiplication of the Fourier transforms. That means that we can perform an inverse
- 00:47:01Fourier transform of these two terms which are just multiplied.
- 00:47:07And this is much faster and computationally much cheaper to produce it in Fourier space.
- 00:47:15So, we're talking about multiplication versus integration so it's very efficient and later
- 00:47:24on you'll figure out that you can perform deconvolution for your light microscopy data.
- 00:47:32So, what is deconvolution you can again imagine the following situation: so you have a test
- 00:47:39image and this test image is multiplied with a Point Spread Function, and Point Spread Function
- 00:47:47is the function which is describing imperfection of your optical system.
- 00:47:52And then, if you zoom in here, you'll see so-called Airy pattern, and the Point
- 00:47:57Spread Function in your light microscope might act as a Gaussian low-pass filter.
- 00:48:05And in the end, if the original object of your interest looks like this,
- 00:48:11with all the high frequency information, the data which we're getting in the microscope will be
- 00:48:17somewhat distorted, so in order to reconstruct the original image, we can apply deconvolution, and
- 00:48:25we can take advantage of the Fourier transforms and this is how this operation is being performed
- 00:48:31in the light microscopy software packages. Somewhat similar operation is performed also
- 00:48:40in cryo-EM, and this is called CTF-correction, and again we're performing two Fourier transforms.
- 00:48:48Now, the second operation which is very much similar to convolution, would be correlation,
- 00:48:58and correlation sometimes they call "similarity measurement",
- 00:49:01and for illustration of the correlation we need to consider again two functions.
- 00:49:07One function will be a cropped area of this image - so something which belongs to this image.
- 00:49:20But if we crop an area like this, it will be very hard to figure out
- 00:49:24where it actually belongs to, because there are many similar areas like that.
- 00:49:28So, to calculate the correlation between these two functions,
- 00:49:36or to compare these two functions in order to find where it matches, what we would have to do in real
- 00:49:44space (would be let's say, for simplicity, it will be like an image or just of four pixels),
- 00:49:52and it will have to go one-by-one pixel in this direction, and to compare whether
- 00:50:00this image belongs to this area or not. And if we are lucky enough, we will compare
- 00:50:06different areas, and then some point we will figure out that the image, the first function
- 00:50:16will be exactly the same as the second function. And we can actually measure measure the value of
- 00:50:23the similarity, and in order to measure the value of such a similarity again, we'd have
- 00:50:28to calculate such a complicated integral. And as you might already guess, if we apply
- 00:50:35for a transform to both sides, instead of integration, we will get multiplication.
- 00:50:43And that means, that in order to calculate the similarity measurement,
- 00:50:47we will be able to take an inverse inverse Fourier transform of the multiplied Fourier
- 00:50:54transforms of each of these functions. And using this approach, we can actually
- 00:50:59calculate the correlation map, so again calculate the correlation map between these two images,
- 00:51:05and we will see something like that, where the bright areas would correspond to high
- 00:51:12correlation and dark areas to low correlation. And the highest value here will correspond
- 00:51:18to the solution of the problem and where the image actually belongs to,
- 00:51:24the cropped area of the image belongs to. So, in cryo-EM, we will apply Fourier transforms
- 00:51:31and the calculations of the correlation, and these values you'll find being
- 00:51:41called cross-correlation coefficients. You'll find these operations in particle
- 00:51:46picking, in image alignments, projection matching and in Fourier Shell Correlation.
- 00:51:52So speaking about Fourier Shell Correlation.. it's the last concept I would like to explain to you.
- 00:52:02To simplify explanation of the Fourier Shell Correlation, which is correlation
- 00:52:06in 3D of the Fourier components, we will consider Fourier Ring Correlation.
- 00:52:12And Fourier Ring Correlation is a correlation of the Fourier components of two images.
- 00:52:23So, what we can do: there are two images, which are not really related - it's not
- 00:52:29the same image. It might have a similar object, but these two images are different.
- 00:52:33So, what we can create, we can create Fourier transforms of each of these images, and then
- 00:52:39if we calculate the Fourier transforms, we will see different amplitude spectra.
- 00:52:44So now, what we can do, we can create or we can draw certain rings of certain diameters,
- 00:52:52and we can we can define these rings in the size, that they would
- 00:53:00cover the whole area of our amplitude spectra. So, they will be a larger and larger with larger
- 00:53:10and larger diameter. And then what we can do, we can pairwise create cross-correlations between
- 00:53:20the Fourier components within these rings. And using such a comparison, which will be dependent
- 00:53:32on the frequency - here is the frequency and here is the correlation value, we will build
- 00:53:38a plot which is called Fourier Ring Correlation. If two images are independent from each other,
- 00:53:43their Fourier components will not be correlated, and this curve will oscillate around zero.
- 00:53:54This are the cross correlation values. In this plot you also see the curve which
- 00:53:59is called "half-bit criterion", which actually reflects the amount of information in all these
- 00:54:06Fourier shells [rings], but it's a bit advanced topic for now. Fourier Ring Correlation of two
- 00:54:14similar objects can be Illustrated in the following example: I took this truck image,
- 00:54:21and then I added noise to this image. The noise will suppress high-frequency details.
- 00:54:26You can see still some low-frequency details, some shape of the car.
- 00:54:32If we calculate the Fourier transform for each of the images, you can see here in
- 00:54:38the amplitude spectrum, that higher frequency components are suppressed.
- 00:54:44If we calculate the Fourier Shell [Ring] Correlation
- 00:54:48between these two images we will see that there is certain values for the cross correlation of
- 00:54:54the lower frequency components and it will slowly be going down to towards the Nyquist frequency.
- 00:55:04And here you can see actually also again the half-bit criterion curve, and the crossing will
- 00:55:11define the information content of this image. So what's important to understand is this cross
- 00:55:19correlation curve, which would reflect the similarity between these images.
- 00:55:26If we generalize 2D case into 3D, so we can easily imagine that during the
- 00:55:36refinement procedure of our single-particle pipeline approach, we can split our data set
- 00:55:44in two halves, and we can determine the 3D reconstruction for each of these halves.
- 00:55:50Then, what we can do, we can create a Fourier transform, 3D-Fourier transform of each of these
- 00:55:57half-maps, and then the same way we would compare shells (this time shells, not the rings) from the
- 00:56:08origin of the image towards the Nyquist frequency. And then we will see, that the lower components of
- 00:56:15the lower Fourier components will correlate higher than the higher frequency components, and this
- 00:56:23plot is called "Fourier Shell Correlation", and where it crosses the criteria for
- 00:56:30determination of the resolution will determine actually the resolution of our reconstruction.
- 00:56:37And it's different compared to X-ray crystallography, where you would have
- 00:56:43resolution as the distance between the origin and the peak of the highest reflection detected.
- 00:56:51In cryo-EM, together with amplitudes we are having also phases and this is the difference.
- 00:56:58So, finally we'll find the natural Fourier transforms which are happening inside your
- 00:57:06microscope, and the objective lenses will be serving as the Fourier transforms of
- 00:57:14your object and in the back focal plane you'll see a diffraction pattern - this would be the
- 00:57:22amplitudes which will correspond to the projection image of your object and the second lens can serve
- 00:57:29as a inverse Fourier transform, and then the image you will be having in the microscope,
- 00:57:34would be an image, which would be actually affected by imperfections of your optical system.
- 00:57:41So, the objective lens in the electron microscope will work as a Fourier Transform.
- 00:57:48So, I've shown you many examples in this lecture, but what I would like you to take with you home
- 00:57:56is understanding that Fourier transform is a mathematical tool which allows you just
- 00:58:06to decompose a function it's a one-dimensional function or an image or three-dimensional function
- 00:58:12as a sum of some other periodic functions and it's just a different representation of your data.
- 00:58:21A different representation, but it's an important representation because it allows you to perform
- 00:58:28certain convenient computational operations. So therefore, Fourier transforms allow you
- 00:58:36to facilitate number of your operations with your data analysis which would be impossible to perform
- 00:58:42in real space, such as correlation, alignments, filtering, resolution measurements and some other.
- 00:58:49It's also important to start using Fourier transform as a diagnostic
- 00:58:54tool of your data analysis as soon as you can and just to get used to that.
- 00:58:59Because each time you're processing your data you can calculate the Fourier transform look
- 00:59:05at the amplitude spectrum and check whether you have some issues with your data or not.
- 00:59:11if you're interested in this topic, I would highly recommend you the lectures from this
- 00:59:18professor from Stanford University - these are the lecturers for electrical engineering
- 00:59:24students and this course is free on YouTube, it's like 30 lectures which are very exciting.
- 00:59:29And two books, which are classic books on the Fourier analysis: from Goodman,
- 00:59:35"Introduction to Fourier Optics" and Bracewell, "The Fourier transform and its applications".
- 00:59:40And with that I would like to thank you for your attention!
- Fourier transforms
- cryo-EM
- data analysis
- sampling
- Nyquist frequency
- image processing
- correlation
- convolution
- filtering
- mathematics