Reference manual for QRI's Oscilleditor

System requirements

The Oscilleditor is designed to be used on a computer. It will probably not work well, or at all, on a smartphone or tablet. But if there's enough interest, we might make a mobile-friendly version.

The Oscilleditor uses a technology called WebGL2 (with two extensions called EXT_color_buffer_float and OES_texture_float_linear) which are not supported on every computer. Specifically, support depends on the graphics card hardware in your computer, the graphics card driver software, and the web browser you use. Sadly we can't provide a list of all models that work or doesn't work - there are too many possible combinations. It has worked on all 10 computers we've tested on before release.

One of the 3 effects in the Oscilleditor (the "Coupled Oscillators" effect) can be quite demanding in terms of computing power. Specifically, the demand increases as you increase the coupling kernel radius, which forces the graphics card to read more and more pixels. If your computer is struggling to keep up, the Frames per Second counter will make that clear. If your computer is struggling but you want a large coupling kernel radius, there is another way to decrease the demand - increase the resolution halvings.

Data flow - the 3 effect types

The Oscilleditor has 3 main types of effects: Pattern Recognition, Coupled Oscillators, and Drifting. This diagram shows how they interact.

For the pattern recognition effect, we input the source photo into Google's InceptionV3 neural network and amplified some of the layers. The resulting images are stored permanently together with the original photos. In other words, this was a one-time job, the neural network doesn't run all the time.

How is the photo an input to the Coupled Oscillators? It depends on what you choose in the GUI. The photo can affect two things: coupling kernel size, and oscillator natural frequency. Either from the photo brightness, depth, or edges (distance to nearest edge).

The original photo is then color blended with the oscillator output and the InceptionV3 output. How much you see of one effect or the other can be controlled by the two "opacity" parameters under Pattern Recognition and Coupled Oscillators Simulation, respectively.

Finally, this blended image is made to wobble by the Drifting effect.

GUI layout overview, main control buttons

Initially, the simulation is paused - you just see a still image. Underneath the image, there are buttons to control this, similar to any video player. Press the Play button to start running the simulation.

If the simulation is already paused, the Pause button is disabled. And vice versa, if the simulation is already playing, the Play button is disabled.

The "Single Step" button advances the simulation one frame forward in time.

Frames per second

Depending on your computer and your parameters, this simulation might be a very heavy task, which your computer might struggle with. After running for 1 second, the "Speed" display will update, it will say something like "60 fps". If it says something much lower, like for example "Speed: 10 fps", it means your computer isn't powerful enough to run the simulation at a smooth speed. If you want to increase the speed, either decrease the coupling kernel radius or increase Resolution Halvings.

Photo selection

Press "Select Photo" to try a different background picture than the default one. This opens a "Choose a Photo" box. The oscilleditor includes 8 photos showing a variety of shapes, textures, natural and manmade scenery. Click one of the 8 small photo thumnails to start using it.

Or supply your own image. Note that this doesn't upload your image to any server on the internet, your image stays locally on your computer. You can choose an image with the Browse button, or you can drag&drop an image file from somewhere else on your computer into the box, or copy an image from anywhere and paste it (Ctrl+V or Cmd+V) while the "Choose a Photo" box is open.

Limitations: when you use your own image, there are no "DeepDream layers", so that part of the Pattern Recognition effect doesn't work. Also there is no "depth map", so the options "Kernel Shrinks By Depth" and "Frequencies Vary By Depth" don't work.

Coupled Oscillators Simulation

This is the most complex effect in the Oscilleditor, with a lot of parameters. The effect runs on a square grid with the same size as the photo, 1280 * 720 pixels. Each pixel is a Kuramoto oscillator which, depending on parameters described further down, wants to synchronize or anti-synchronize with other nearby oscillators.

Opacity

Opacity controls how the original photo and the oscillator effect are mixed together, as a fraction between 0 and 1, where 0 means that the oscillators are completely invisible. 1 means that they take over and completely hide the underlying photo.

Colors

Since each pixel is fundamentally a Kuramoto oscillator, which spins through 360 degrees, how should its current angle be visualized on the screen? This dropdown lets you choose between a bunch of coloring schemes - they decide how an angle maps to a color.

"Rainbow" simply maps each angle to a hue around the color wheel, so that 0 degrees is pure red, 60 is pure yellow, 120 is pure green, and so on up to 359 degrees.

"Rainbow double" is the same but with two full rainbows in one single oscillation. In other words, 0 and 180 are both red, and X+180 is the same color as X.

"Black & white smooth, normal blending" maps the angle to a grayscale gradient between black and white, so 0 is white and 180 is black. "Normal blending" means that if you have two layers of oscillators, the final color is a weighted average of the colors from the two layers.

"Black & white smooth, interference blending" - same colors, but with a different way to blend between two layers of oscillators. It's not really related to wave interference, we should choose a better name.

"Black & white sharp, interference blending" - no grayscale gradient, just black and white like 60's op art. The final color is white if exactly one of the layers is white, but not if both of them are. In computer programming this is known as the "XOR" or "exclusive or" operation.

"Complementary colors sharp, normal blending" maps the angle (in oscillator layer 1) to a color gradient between blue at 0 degrees and yellow at 359 degrees, then immediately after yellow we jump straight back to blue, so the gradient is one-directional, it doesn't go back and forth. But in oscillator layer 2, the color gradient instead goes from green to magenta.

"Complementary colors sine, asymmetric" - the angle from layer 2 maps to a gradient between two complementary colors on opposite sides of the color wheel. The angle from layer 1 moves that layer 2 gradient partway around the color wheel - wobbles it back and forth between 23 and 157 degrees. So this is an asymmetric effect - the two layers don't work the same way.

"Complementary colors rotate, asymmetric" - the angle from layer 2 maps to a gradient between two complementary colors on opposite sides of the color wheel. The angle from layer 1 rotates that gradient all around the color wheel. So this is an asymmetric effect - the two layers don't work the same way. In this mode, the blending balance parameter has no effect.

Num layers

You can have 1 or 2 "layers" of oscillator simulation running in parallel. By default, the two layers are independent and don't affect each other, but you can make them affect each other with the cross-layer coupling parameter, which can give rise to very complex patterns. If you just have 1 layer, all the controls relating to layer 2 will get disabled.

Blending Balance

If you have two layers of oscillators, this controls how they get blended together. When the slider is close to 1, that layer is most visible. Around 1.5, both layers are equally visible. And when the slider is close to 2, that layer is most visible.

Layer 1 and Layer 2

Each of the two layers has an identical set of controls.

Log-Polar Transform

This checkbox can be on or off. When off, the oscillators run on a standard square grid, aka cartesian plane. When on, we switch to a log-polar coordinate system, which essentially measures distances from the origin (center of image) and angle to the origin. This feature is inspired by previous research about the mapping from the retina to the early visual cortex, and can be used to generate so-called "form constants" like cobwebs and flowers.

To be precise, our distance mapping is ln(0.2 + distance * 23) / pi, where distance is not measured in pixels but in a different unit, scaled uniformly so the grid is 1 unit high and 1.7777... units wide (because the grid aspect ratio is always 16:9). So distance can vary between 0 (in the center of the image) and sqrt(0.5^2+0.888^2) = 1.02 (at a corner of the image). The constants are somewhat arbitrary but have been tweaked to find the best compromise between two opposing goals: have the result stay mostly within the range 0-1, and minimize distortion near the center (where a plain log function would go towards negative infinity).

Resolution Halvings

This feature was created as a way to increase simulation speed on weaker computers with large kernel radius. As described above, the simulation runs on a 1280 * 720 pixel grid. If your kernel radius is 10, that means the kernel has a diameter of 20, and the whole kernel is a 20*20 square. So for each pixel in the grid, we also read 400 neighboring pixels that the kernel covers, for a total of 1280 * 720 * 20 * 20 = over 368 million texture reads. This is true when "resolution halvings" is 0. But if we increase it to 1, the grid resolution is halved in both x and y, which reduces the number of pixels to one fourth, which saves a lot of computing power. The main disadvantage is that the image simulation gets blockier, until a pixel is bigger than the coupling kernel (or the kernel is smaller than a pixel, depending on how you think about it). The following pictures illustrate the logic of what happens. They are not directly representative of what you will see inside the actual tool.

In practice, in the tool, the patterns will look less blocky even with Resolution Halvings > 0, because we smooth them out before display. So this is what 3 would look like:

And this is what 4 would look like:

Kernel Radius

This scales the whole coupling kernel uniformly. It sets the outer radius in pixels of the kernel. (The distance numbers in the "Coupling kernel shape controls" table are relative fractions of this outer radius.) Specifically, it sets the maximum radius. Because the kernel can be smaller in some parts of the image, if you choose to set that option under Kernel Shrinks By, and the Kernel Shrink Factor number is a relative fraction of this maximum radius.

As you change the Kernel Radius, the grayscale preview in the top-left corner will resize in real time to show you how big the kernel is visually. If the simulation is playing, this preview will disappear automatically after 2 seconds. If the simulation is paused, this preview will disappear next time you change any other parameter that's not related to the kernel, or next time you Play or Single Step or so.

Kernel Shrinks By

By default, the kernel is the same size all over the image. But if you change this option, the kernel size can vary depending on the local image content, leading to interesting effects.

Many people report that psychedelic patterns tend to emenate from edges or get attracted by edges, so therefore "kernel shrinks by edges" is one of our options. The other is "by depth".

As you change this option, we illustrate what it will do with differently-scaled circles on the image.

If the simulation is playing, this preview will disappear automatically after 2 seconds. If the simulation is paused, this preview will disappear next time you change any other parameter that's not related to the kernel scaling, or next time you Play or Single Step or so.

So as you see, we are providing the simulation with depth and edges as input. Obviously, the brain itself doesn't do that, it finds edges on its own and derives depth based on several clues. As we develop more sophisticated models in the future, we hope to make these effects emerge naturally out of the model itself.

Kernel Shrink Factor

Controls how much the kernel should change radius based on the shrinks by option. Fraction between 0 and 1, where 1 means it doesn't shrink at all. 0.5 means that exactly on an edge, where the kernel wants to shrink maximally, it will shrink to 50% of its max radius.

Coupling kernel shape controls

Our coupling kernel consists of 4 concentric rings.

This table of controls might look daunting to understand.

The controls allow you to control each of the 4 rings individually. "Distance" moves each ring in or out from the center, so 0 is exactly at the center, and 1 is all the way at the edge. You'll want to avoid having a ring all the way at the edge, because it will get abruplty clipped there, leading to artifacts in the simulation. If you want a larger ring, scale up the whole kernel with the kernel radius parameter.

"Width" makes each ring wider or narrower. Note that wider also means smoother, and narrower also means sharper.

Finally the most important value, "Coupling", decides what each ring means. A coupling number above 0 means that each oscillator will try to synchronize its oscillations with its neighbors that touch the ring, if we put the whole kernel centered on the oscillator in question. This positive coupling is displayed as white color in the diagram. The value also controls the strength of the coupling - the higher the number, the more I'll care about synchronizing with those neighbors. Coupling 0 means neutral, nothing in particular happens, which is displayed with gray. And below 0 means negative coupling, so each oscillator will try to anti-synchronize with its neighbors that touch the ring. This means they try to be maximally out of phase - if my neighbors are 10 degrees ahead of me, I'll slow down to try to make them 180 degrees ahead of me.

If you put one white and one black ring on top of each other, with the same width, they cancel out.

As you change these coupling controls, the grayscale preview in the top-left corner will update in real time. If the simulation is playing, this preview will disappear automatically after 2 seconds. If the simulation is paused, this preview will disappear next time you change any other parameter that's not related to the kernel, or next time you Play or Single Step or so.

You might wonder what happens near the edges and corners of the simulation grid - how do we handle it when the kernel extends outside the grid? We use coordinate "wraparound" like in early videogames, so when the kernel extends above the top edge of the grid, we instead sample near the bottom edge. Topologically, that means the grid forms a torus / donut shape. We're not claiming that's what happens in the brain, it's just a convenient way to solve the problem in the simulation.

Coupling to Layer 2 / to Layer 1

If you have Num Layers = 2, by default the two layers are independent and don't affect each other, but you can make them affect each other with this parameter. This is a coupling number just like for the 4 rings above, so above 0 means positive coupling and so on. It will make each oscillator in one layer want to synchronize with the oscillator at the exact same place in the other layer.