Fityk - User's Manual

The window of fityk program consists of (from the top): menu bar, toolbar, main plot, auxiliary plot, output window, input field, status bar and of sidebar at right-hand side. The input field allows to type and execute commands in similar way as it is done in CLI version. The output window (which is configurable through pop-up menu) shows results. Actually all GUI commands are converted into text and visible in output window.

The main plot can display data points, functions and/or the sum of all functions. Use pop-up menu (click right button on the plot) to configure it. Some properties of plot (eg. colors of data points) can be changed using the sidebar.

One of the most useful things which auxiliary plot can display is the difference between data and the sum of functions. The plot can be controlled by pop-up menu. I hope quick look at this menu and a minute of experiments will make all possibilities of the auxiliary plot clear.

Sometimes it is useful to print effects of our fitting work. Hard copy will contain plots visible on the screen, scaled to fit the page. The only difference is that backgrounds of plots will be white (to not waste toner/ink). So it may be necessary to change color of axis or data to darker one.

Configuration of GUI (visible windows, colors, etc.) can be saved using GUI->Save current config. Two different configurations can be saved, what allows easy changing of colors for printing. On Unix platform, these configurations are stored in file in user's home directory. On Windows - they are stored in registry (perhaps in future they will be also stored in file).

The usage of the mouse on menu, dialog windows, input field and output window is intuitive, so the only topic described here is how to effectively operate mouse on plots.

Let us start with the auxiliary plot. Right button displays pop-up menu, with left button you can select range to be displayed - range on x axis. Clicking with middle button (or with left button and pressed Shift) will zoom it out and display all data.

On the main plot, meaning of the left and right mouse button depends on current mode, that can be changed using toolbar or menu. There are hints on the status bar. In normal mode, left button is used for zooming and right invokes pop-up menu. The same behaviour can be obtained in any mode by pressing Ctrl (or Alt or Shift). Middle button can be used to select a rectangle that you want to zoom in. If an operation has two steps, like rectangle zooming (first you press button to select first corner, then you move mouse and release button to select second corner of rectangle), you can cancel it by pressing another button when first one is pressed.

Data are stored in files. Unfortunately, there are various formats of files with data. The basic one is text file with every line corresponding to one data point. The line should contain at least two numbers: x and y of point. It can also contain standard deviation of y coordinate. Numbers can be separated by whitespace, commas, colons or semicolons. Some lines can contain comments or extra informations. If these lines have a hash (#) in first column, they are ignored. In other case, they are also ignored (unless they can be read as data point). There are also other file types, that can be read: .rit, .cpi, Siemens-Bruker.raw and .mca. In future, the way the special file formats are handled will be changed (external library will be used for it, unfortunatelly this library does not exists yet).

Points are loaded from files using command

where dataslot should be replaced with @0, unless many datasets are to be used simultanously (for details see: the section called “Working with many datasets”), filetype can be omitted (at this moment, due to a small number of supported formats, the filetype can be detected automatically), xcol, ycol, scol, are unsigned integers. If the filename contains blank characters, semicolon or comma, it should be put inside of single quotation marks. If the file is in a text format (columns with numbers) it can be specified which column contains x, y and, optionally, std. dev. of y.

Command

can export data to 3-column (x, y and standard deviation) ASCII file. Only active points are being exported (see next chapter to learn about active and inactive points).

Some information about current data can be obtained using command:

We often have situation, that only part of data from file is interesting for us. We should be able to exclude selected points from fitting and all computations. Every point can be either active or inactive. It can be done with command A=... (see the section called “Data transformations” for details) or with mouse-click in GUI. The idea of active and inactive points is simple: only the active ones are subject to fitting and peak-finding, inactive ones are neglected in these cases.

When fitting data, we assume that only y coordinate of data is subject to statistical errors in measurement. It is very common assumption. To see how y standard deviation influences fitting (optimization), look at weighted sum of squared residuals formula in the section called “Nonlinear optimization ”. We can also think about weights of points - every point has a weight assigned, that is equal

Standard deviation of points can be read from file together with x and y coordinates. Otherwise, it is set to max(sqrt(y), 1.0), Setting std. dev. as a square root of value is common and has theoretical ground when y is the number of independent events. You can always change standard deviation, eg. make it equal for every point with command: S=1. See the section called “Data transformations” for details.

Every data point has four properties: x coordinate, y coordinate, standard deviation of y and active/inactive flag. Lower case letters x, y, s, a stand for these properties before transformation, and upper case X, Y, S, A for the same properties after transformation. M stands for the number of points. You can transform data using assignments. Command Y=-y will change the sign of y coordinate of every point. You can also apply transformation to selected points: Y[3]=1.2 will change point with index 3 (which is 4th point, because first has index 0), and Y[3...6]=1.2 will do the same for points with indices 3, 4, 5, but not 6. Y[2...]=1.2 will apply transformation to point with index 2 and all next points. You can guess what Y[...6]=1.2 does. Most of operations are executed sequentially for points from the first to the last one. n stands for the index of currently transformed point. The sequance of commands: M=500; x=n/100; y=sin(x) will generate the sinusoid dataset with 500 points.

Points are kept sorted according to x coordinate, so changing x coordinate of points will also change the order and indices of points.

Expressions can contain real numbers in normal or scientific format (eg. 1.23e5), constant pi, binary operators: +, -, *, /, ^, one argument functions: sqrt, exp, log10, ln, sin, cos, tan, atan, asin, acos, abs, round (rounds to the nearest integer), two arguments functions: min2, max2 (eg. max2(3,5) will give 5), and ternary ?: operator: condition ? expression1 : expression2, which performs expression1 if condition is true and expression2 otherwise. Conditions can be built using boolean operators comparisions: AND, OR, NOT, >, >=, <, <=, = (or ==), != (or <>), TRUE, FALSE.

Important note: all operations are performed on real numbers. Two numbers that differ less than epsilon=1e-9 ie. abs(a-b)<epsilon, are considered equal. Indices are also computed in real number domain, and then rounded to nearest integer.

Transformations can be joined with ampersand (&), eg. X=y&Y=x swaps axes.

Before and after executing transformations, points are always sorted according to x coordinate. You can change order of points using order=t, where t is one of x, y, s, a, -x, -y, -s, -a. It only has a sense in sequence of transformations joined with &, because after finishing transformation, points will be reordered again.

Points can be deleted using following syntax: delete[index-or-range] or delete(condition) and created simply by increasing value of M.

There are two parametrized functions: spline and interpolate. The general syntax is: parametrizedfunc [param1, param2](expression) eg. spline[22.1, 37.9, 48.1, 17.2, 93.0, 20.7](x) will give value of cubic spline interpolation through points (22.1, 37.9), (48.1, 17.2), ... in x. Function interpolation is similar, but give polyline interpolation. Spline function is used for manual background substraction in GUI.

A few examples:

     
     Y[1...] = Y[n-1] + y[n] # integrate

     x[...-1] = (x[n]+x[n+1])/2;  # reduces
     y[...-1] = y[n]+y[n+1];      # two times
     delete(n%2==1)               # number of points

     delete(not a) # delete inactive points

     X = 4*pi * sin(x/2*pi/180) / 1.54051 # changes x scale (2theta -> Q)
     

Introduced later in this manual variables also can be used in data transformations, eg. Y=y/$foo.

Let call a set of data that usually comes from one file - a dataset. All operations described above assume that only one dataset. If there are more datasets created, it must be explicitly written which dataset the command is applied to, eg. M=500 in @0. Datasets have numbers and are referenced by '@' with the number, eg. @3. @* means all datasets, and Y=y/10 in @* will do what is expected to.

Command

will load dataset to new slot. Using @+ increases the number of datasets, and command delete @n decreases it. It is also possible to duplicate dataset (command @+ < @n) or create new dataset as a sum of two or more existing ones (command @+ < @n + @m + ...).

Each dataset has a separate sum, ie. a model that can be fitted to the data. It is explained in the next chapter.

Each dataset has a title. It does not have to be unique. When loading file, a title is automatically created, either using filename or it is read from the file (it depends on the format of the file). It is used for exporting data -- some file formats require it. Title can be changed using command @n.title=new-title . To see title of the dataset, use info @n.

The sum of functions S - curve that is fitted to data - is itself a function. The value of the whole sum is computed as a sum of functions, like Gaussians or polynomials, and can be given by formula: , where f_i is a function of x, and depends on a vector of parameters a. This vector contains all fitted parameters. Because we often have the situation, that the error in the x coordinate of data points can be modeled with function z(x; a), we introduce this term to sum:

where . Note that the same z(x) is used in all functions.

Now we will have a closer look at f_i functions. Every function f_i has a type chosen from function types available in the program. The same is true about functions z_i. One of these types is the Gaussian. It has the following formula:

There are three parameters of Gaussian. These parameters does not depend on x. There must be one variable binded to each parameter.

Variables in Fityk have names prefixed with dollar ($). Variable is created by assigning a value to it, eg. $foo=~5.3 or $c=3.1 or $bar=5*sin($foo). $foo is here a so-called simple variable. It is created by assigning to it real number prefixed with ~. The `~' means that value assigned to the variable can be changed when fitting sum to data. For people familiar with optimization techniques: the number of defined simple variables is the number of dimensions of space we are looking for optimum in. Variable $c is actually a constant. $bar depends on the value of $foo. When $foo changes, the value of $bar also changes. Compound variables can be build using operators +, -, *, /, ^ and functions sqrt, exp, log10, ln, sin, cos, tan, atan, asin, acos.

Every simple parameter has a value and, optionally, domain. Domain is used only by fitting algorithms, which need to randomly initialize or change variables. Genetic Algorithms are a good example. [TODO: setting domain is not implemented at this moment, but will be added soon]

Variables can be used in data tranformations. Also value of data expression can be used in variable definition, but it must be inside of braces, eg. $bleh={M} or, to create simple variable: $bleh=~{M}.

Sometimes it is useful to freeze variable, ie. to prevent it's changes in fitting. There is no special syntax for it, but it can be done using data expressions in this way:

      $a = ~12.3 # $a is fittable
      $a = {$a}  # $a is not fittable
      $a = ~{$a}  # $a is fittable again
     

It is also possible to define a variable as eg. $bleh=~9.1*exp(~2). In this case two simple variables (with values 9.1 and 2) will be created automatically. Automatically created variables are named $_1, $_2, $_3, etc.

Variables can be deleted using command delete $variable.

Let's go back to functions. Function types have names that start with upper case letter, eg. Linear or Voigt. Functions (ie. function instances) have names prefixed with percent, eg. %func. Every function has a type and variables binded to its parameters.

To see list of available function types, use command info types. You can also use command info typename, eg. info Pearson7 to see parameter names, default values and formula.

Function can be created by giving type and a proper number of comma-separated variables in brackets, eg: %f = Gaussian(~66254., ~24.7, ~0.264) or %f = Gaussian(~66254., $ctr, $b+$c). Every expression, which is valid on the right hand side of variable assignment, can be put as a variable. If it is not just a name of a variable, an automatic variable is created. In the last example two variables are created (value 66354 and the sum).

The second way is to give named parameters of function, in any order, eg. %f = Gaussian(height=~66254., hwhm=~0.264, center=~24.7) Function types can can have specified default values for some parameters, so this assignment is also valid: %f = Pearson7(height=~66254., center=~24.7, fwhm=~0.264) , although shape parameter of Pearson7 is not given.

A deep copy of function (ie. all variables it dependends on are copied) can be made using command %function =copy(%anotherfunction)

Functions can be also created with command guess, as described in the section called “Guessing peak location ”.

You can change the variable binded to any of function parameters in this way:

      =-> %f = Pearson7(height=~66254., center=~24.7, fwhm=~0.264)
      New function %f was created.
      =-> %f[center]=~24.8
      =-> $h = ~66254
      =-> %f[height]=$h
      =-> info %f
      %f = Pearson7($h, $_5, $_3, $_4)
      =-> $h = ~60000 # variables are kept by name, so this also changes %f
     

Functions can be deleted using command delete %function.

With default settings, value of every function is calculated in every point. Functions like Gaussian often have non-neglectible value only in small fraction of all points. To speed up calculation, set option cut-function-level to non-zero value. Note, that some functions may not support this optimization at all, and for other approximations are used and the exact value of cut-off level can differ.

As it was already written, each dataset has a separate sum. ie. a model that can be fitted to the data. It can be seen in formula above, that the sum consists of functions f_i and z_i. Each dataset has two sets named F and Z. They contain names of functions. Sum is constructed by telling which functions are in F and which in Z.

In many cases Z can be forgotten. Then fitted curve is a sum of all functions in F. The functions can be listed with command info F.

Command %function -> F puts %function into F, command %function -> Z puts %function into Z, and command %function -> N removes %function from F or Z. If there is more than one dataset, F, Z and N must be prefixed with dataset number, eg. %function -> @1.F or %function -> @0.N . Following syntax is also valid:

  # create and add funtion to F
  %g = Gaussian(height=~66254., hwhm=~0.264, center=~24.7) -> @0.F
  # create automatically named funtion and add it to F
  Gaussian(height=~66254., hwhm=~0.264, center=~24.7) -> @0.F
  # clear F
  @0.F = 0
  # clear F and put three functions in it
  @0.F = %a, %b, %c
  # make @1.F the exact (shallow) copy of @0.F
  @1.F = @0.F
  # make @1.F a deep copy of @0.F (it means all functions and variables
  # are duplicated).
  @1.F = copy(@0.F) 

Sum can be exported as data points (.dat), mathematic formula (.formula) or peak parameters (.peaks) using command: F > filename, where filename have one of extensions listed above.

It is often required to keep width or shape of peaks constant for all peaks in dataset. To change variables binded to parameters with given name of all functions in F, use command: F[param]=variable . Examples:

  F[hwhm]=$foo # hwhm's of all functions in F that have parameter hwhm will be 
               # equal $foo. (hwhm means here half-width-at-half-maximum)
  F[shape]=%_1[shape]  # variable binded to shape of peak %_1 is binded
                       # also to shapes of all functions in F
  F[hwhm]=~0.2  # For every function in F a new variable is created and binded 
                # to parameter hwhm. All these parameters are independent. 

It is possible to guess peak location and add it to F with command: %name = guess PeakType [x1:x2] in @n , eg. guess Gaussian [22.1:30.5] in @0. If the range is omitted, the whole dataset will be searched. Name of function is optional.

Fityk offers only primitive algorithm for peak-detection. It looks for highest point in given range, and than tries to find out width of peak.

If the highest point is found near the boundary of the given range, it is very probable that it is not the peak top, and, if the option can-cancel-guess is set true, the guess is canceled.

There are two real-number options related to guess: height-correction and width-correction. Default value of them is 1. The guessed height and width are multiplied by the values of these options respectively.

If you are using GUI, most of informations can be displayed with mouse clicks. Otherwise, you can use info command. Using info+ instead of info sometimes displays more datailed informations. Command info peaks range will show, where the guess command would find the peak. info functions lists all defined functions, and info variables - all variables. info @n.F and info @n.Z show informations about F and Z, info @n.formula shows mathematic formula of fitted function, and info @n.dF(x) compares symbolic and numeric derivatives in x (useful only for debugging).

This is the core. We have a set of observations (data points), and we want to fit a model (sum of functions), that depends on adjustable parameters, to observations. Let me quote Numerical Recipes, chapter 15.0, page 656 (if you do not know the book, visit http://www.nr.com ):

Our function of merit is WSSR - weighted sum of squared residuals, called also chi-square:

Weights can be are based on standard deviations, . You can learn why squares of residuals are minimized eg. from chapter 15.1 of Numerical Recipes. So we are looking for a global minimum of chi². This large field of numerical research - looking for minimum or maximum - is usually called optimization; it is non-linear and global optimization. Fityk implements three very different optimization methods. All are well-known and described in many book.

To fit sum to data, use command

Plus (+) means that fitting method is not initialized. It is used to continue previous fitting. All non-linear fitting methods are iterative. number-of-iterations is the maximum number of iterations. There are also other stopping criteria, so the number of executed iterations can be smaller.

Fitting methods can be set using set command: set fitting-method = method, where method is one of: Levenberg-Marquardt, Nelder-Mead-simplex, Genetic-Algorithms.

All non-linear fitting methods are iterative, and there are two common stopping criteria. The first is the number of iterations. If it is not specified with command, value of option default-max-iterations is used. And the second is the number of evaluations of objective function (WSSR), specified by value of option max-wssr-evaluations. It is approximately proportional to time of computations, because most of time in fitting process is taken by evaluating WSSR. There are also other criteria, different for each method. Note that values of all options, even those common for all methods, are set separately for every method.

If you give too small n to f.run command, and fit is stopped because of it, not because of convergence, it makes sense to use f.continue command to process further iterations. [TODO: how to stop fit interactively]

When fit is running, each iteration outputs some informations. If they are scrolling too fast, you can reduce it n times by assigning n to output-one-of option.

Setting o.set autoplot = on-fit-iteration will draw a plot after every iteration, to visualize progress. (see autoplot)

Command ^[1] :

can be used to display WSSR, symmetric errors and variance-covariance matrix.

Available methods can be mixed together, eg. it is sensible to obtain initial parameter estimates using simplex method, and than fit it using Levenberg-Marquard method. Command s.history can be useful for trying various methods with different options and/or initial parameters and choosing the best solution.

Some fitting methods are using random number generator. In some situations one may want to have repeatable and predictable results of fitting, eg. to make a presentation. Seed for a new sequence of pseudo-random numbers is set at the beginning of fitting initialization (when f.run is called). If value of pseudo-random-seed option is set to -1, the seed is based on system time and sequence of pseudo-random numbers is every time different. In other case, if pseudo-random-seed option has a non-negative value, this value is used as a seed. Remember, that like all other options, value of pseudo-random-seed is independent for each fitting method.

It is a standard of nonlinear least-squares routines. It involves computing first derivatives of functions. For description of L-M method see Numerical Recipes, chapter 15.5 or Siegmund Brandt Data Analysis, chapter 10.15 (I've read the Polish translation of the second). In a few words: it combines inverse-Hessian method (called Gauss-Newton method?) with steepest descent method by introducing lambda factor. When lambda is equal 0, the method is equivalent to inverse-Hessian method. When lambda increases, the shift vector is rotated toward the direction of steepest descent and the length of the shift vector decreases. (The shift vector is a vector that is added to the parameter vector.) If the better fit is found in iteration, lambda is decreased - it is divided by value of lambda-down-factor option (default: 10). Otherwise, lambda is multiplied by value of lambda-up-factor (default: 10). You can also change a value of lambda-starting-value option.

Marquardt method has one stopping criterium apart from common stopping criteria. If it happens two times in sequence, that relative progress in value of objective function (WSSR) is smaller then value of stop-rel-change option, fit is considered to be converged and is stopped.

L-M method finds a minimum quickly. The question is, if it is the global minimum. It can be a good idea to add a small random vector to the vector of parameters and try again. This small shift vector is added, when value of shake-before option is positive (by default it is 0). Value of every parameter's shift is independent and randomly drawn from distribution of type specified by value of shake-type option (see option distrib-type) in simplex method). The expected value of parameter shift is directly proportional to both value of shake-before option and width of parameter's domain.

This time I am quoting chapter 4.8.3, p. 86 of Peter Gans Data Fitting in the Chemical Sciences by the Method of Least Squares

This method is also described in previously mentioned Numerical Recipes (chapter 10.4) and Data Analysis (chapter 10.8).

There are a few options for tuning this method. One of those is a stopping criterium min-fract-range. If value of expression 2(M-m)/(M+m), where M and m are values of worst and best vertices respectively (values of objective functions of vertices, to be precise), is smaller then value of min-fract-range option, fitting is stopped. In other words, it is stopped if all vertices are at almost the same level.

The rest of options is related to initialization of simplex. Before starting iterations, we have to chose set of points in space of parameters, called vertices. Unless option move-all is set, one of these points will be the current point - values that parameters have at this moment. All but this one are drawn as follows: each parameter of each vertex is drawn separately. It is drawn from distribution, that has center in center of domain of parameter, and width proportional to both width of domain and value of move-multiplier parameter. Distribution type can be set using option distrib-type as one of: uniform, Gaussian, Lorentzian and bound. The last one causes value of parameter to be either greatest or smallest value in domain of parameter - one of two bounds of domain (assuming that move-multiplier is equal 1).

[TODO]

In GUI version there is hardly ever need to use this command directly.

Command plot controls visualization of data and the sum. It is used to plot given area - in GUI it is plotted in program's main window, in CLI popular program gnuplot is used, if available.

xrange and yrange have one of two following syntaxes:

The second is just a dot (.), and it remains appropriate range not changed.

Examples:

   plot [20.4:50] [10:20] #plot will show x from 20.4 to 50 and y from 10 to 20

   plot [20.4:] # x from 20.4 to the end, 
                # y range will be fitted to contain all data

   plot . [:10] # x range will not be changed, y from the lowest point to 10 
   plot [:] [:] # all data will be showed   
   plot         # all data will be showed   
   plot . .     # nothing changes
     

Value of option autoplot changes automatic plotting behaviour. By default, plot is refreshed automatically after changing the data or the sum of functions. It is also possible to visualize every iteration of fitting method by replotting peaks after every iteration.

First, there is an option verbosity (not related to command info) which decides about amount of messages displayed when executing commands.

If you are using GUI, most of informations can be displayed with mouse clicks. Otherwise, you can use info command. Using info+ instead of info sometimes displays more datailed informations.

[TODO: list of all info arguments]

All commands given during program execution are stored in memory. They can be listed using command: commands [n:m] or written to file: commands [n:m] > filename .

To log commands to file, when they are executed, use: commands > filename or, to log also output: commands+ > filename . To stop logging, use: commands > /dev/null .

Scripts can be executed using command: commands < filename .

There is also a command dump > filename, [TODO] which is not working at this moment.

Command sleep sec makes program waiting sec seconds, doing nothing.

Command quit works as expected.

If option exit-on-warning is set, any warning will close the program. It ensures, that no warnings can be overlooked.