9.3.4: The `convert' commands

• Convert Columns To Grid: Create grid from (x,y,f) data
• Convert Columns To Spline: Create spline (x',y') from (x,y) data
• Convert Grid To Columns: Create (x,y,f) data from grid
• Convert Grid To Image: Create an image from grid data
• Convert Image To Grid: Create a grid from image data

9.3.4.1: `convert columns to grid'

`convert columns to grid OPTIONS'

where the `OPTIONS' may be omitted or selected from this list:

`neighbor' `boxcar [.xr. .yr. [.n. .e.]]' `objective [.xr. .yr. [.n. .e.]]' `barnes [.xr. .yr. .gamma. .iter.]'

For more discussion on the methods see see Ungridded Data.

All these commands ``grid'' columnar (x,y,z) data. That is, they fill up a grid based on some form of interpolation of the possibly randomly-spaced columnar data. There are many methods in existence for doing this, and Gri implements several of them as alternatives.

The grid will have been defined by commands such as `set x grid', `set y grid', `read grid x' and `read grid y'. As of version 2.1.9, Gri does not require a grid to have been pre-defined; it will create a regular 20 by 20 grid, spanning the range of x and y data, as a default. This is a good starting point in many cases.

`neighbor' method

Very fast but very limited.

`boxcar' method

Slower but a lot better. Still, this can produce noisy contours if the data are not densely and uniformly ditributed through domain.

`objective' method

Somewhat slower than `boxcar', but produces better fields since the averaging function is smooth.

`barnes' method

Somewhat slower than `objective', but only by a constant factor (that is, independent of number of data). This produces by far the best results, since the smoothing function has variable spatial scale. This is the default method if no method is supplied.

The default `.xr.' and `.yr.' are calculated by determining the span in x and in y directions, and dividing each by the square root of the number of data points. These numbers are then multiplied by the square root of 2. The method is as proposed by S. E. Koch and M. DesJardins and P. J. Kocin, 1983. ``An interactive Barnes objective map anlaysis scheme for use with satellite and conventional data,'', J. Climate Appl. Met., vol 22, p. 1487-1503.

If `.xr.' and `.yr.' were supplied but negative, then Gri interprets this as an instruction to modify the default values, described in last paragraph, by multiplying by the absolute values of the negative numbers given, instead of muliplying by square root of 2.

If the `chatty' option is turned on then Gri will print out the values of (dx,dy) that it has calculated; this gives you some guidance for supplying your own values of `(.xr.,.yr.)' if you choose to supply them yourself. It is also a good idea to let these parameters be a guide for your grid spacing; for example, Koch et al., 1983, suggest using grid spacing of 0.3 to 0.5 times (dx,dy).

And now, the details ...

• ``Neighbor'' method The `convert columns to grid neighbor' method is useful for (x,y,z) data which are already gridded (i.e., for which x and y take only values which lie on the grid), or nearly gridded. The (x,y,z) data are scanned from start to finish. For each data point, the nearest grid point is found. Nearness is measured as Cartesian distance, with scale factor given by the distance between the first and second grid points. In other words, distance is given by D=sqrt(dx*dx+dy*dy) where dx is ratio of distance from data point to nearest grid point, in x-units, divided by the difference between the first two elements of the x-grid, and dy is similarly defined. Once the grid point nearest the data point is determined, Gri adds the z-value to a list of possible values to store in the grid. Once the entire data set has been scanned, Gri then goes back to each grid point, and chooses the z-value of the data point that was nearest to the grid point -- that is, it stores the z value of the (x,y,z) data triplet which has minimal D value. Note that this scheme is independent of the order of the data within the columns.

  The `neighbor' method is useful when the data are already pre-gridded, meaning that the (x,y,z) triplets have x and y values which are already aligned with the grid. Computational cost: For `P' data points, `X' x-grid points, and `Y' y-grid points, the method calculation cost is proportional to `P*[log2(X)+log2(Y)]' where `log2' is logarithm base 2. As discussed below, this is often several orders of magnitude lower than the other methods of gridding.

• ``Objective'' method In the `objective' method, a smoothing technique known as objective mapping is applied. It is essentially a variable-size smoothing filter of approximately Gaussian shape (it is method ``two'' of Levy and Brown [1986 J. Geophysical Res. vol 91, p 5153-5158]) The parameters `.xr.' and `.yr.' give the width of the filter.

  With the optional additional parameters `.n.' and `.e.' are specified, then grid values will be assigned the missing value if there are fewer than `.n.' (x,y,f) data in the neighborhood of the gridpoint, even after enlarging the neighborhood by widening and heightening by root(2) up to `.e.' times. (The enlargement is only done if fewer than `.n.' points are found.) If these parameters are not specified in the command, then values `.n.'=5 and `.e.'=1 are assumed. The special case where `.e.' is negative tells Gri to always fill in each grid point, by extending the neighborhood to enclose the entire dataset if necessary.

  Computational cost: For `P' data points, `X' x-grid points, and `Y' y-grid points, the method calculation cost is proportional to `P*X*Y'. Given that `X' and `Y' are determined by the requirement for smoothness of contours and the size of the graph, they are more or less fixed for all applications. They are often in the range of 20 or so -- on 10 cm wide graph, this yields a contour footprint of 1/2 cm, which is often small enough to yield smooth contours. Therefore, the computational cost scales linearly with the number of data points. Compared to the ``neighborhood'' method, this is more costly by a factor of `X*Y/log_2(X)/log_2(Y)' which is normally in the range from 20 to 50.

• ``Boxcar'' method In the `boxcar' method, the grid points are derived from simple averages calculated in rectangles `.xr.' wide and `.yr.' tall, centred on the gridpoints. The `.n.' and `.e.' parameters have similar meanings as in the ``objective'' method.

  Computational cost: Roughly same as `objective' method described above.

• ``Barnes'' method This is the default scheme.

  The Barnes algorithm is applied. If no parameters are specified, `.xr.' and `.yr.' are determined as above, with `.gamma.' set to 0.5, and `.iter.' set to 2 so that two iterations are done. On successive iterations, the smoothing lengthscales `.xr' and `.yr' are each reduced by multiplying by the square root of `.gamma.'. Smaller `.gamma.' values yield better resolution of small-scale features on successive iterations. Koch et al., 1983, recommend using a `.gamma.' value in the range 0.2 to 1, with two iterations.

  Provided that all the grid points are close enough to at least some column data, the entire grid is filled. But if `.xr.' and `.yr.' are too small, the weighting function can fall to zero, since it is exponential in the sum of the squares of the x-distance/`.xr.' and the y-distance/`.yr.'; in that case missing values result at those grid points. On a 32 bit computer, the weighting function will fall to zero when x-distance/`.xr.' and y-distance/`.yr.' are less than about 15 to 20.

  If weights have been read in (see Read Columns), then these values are applied in addition to the distance-based weighting. (The normalization means that weights for two data points of e.g. 1 and 2 will yield the same result as if the weights had been given as 10 and 20.)

  The computational cost at each iteration scales as `P*X*Y)'. This is comparable to that of the ``objective'' and ``boxcar'' methods. Since normally two iterations are done, ``barnes'' is about double the cost of these methods. (Note: versions prior to 2.1.8 were much slower for large datasets, being proportional to `P*P'.)

  References: (1) Section 3.6 in Roger Daley, 1991, ``Atmospheric data analysis,'' Cambridge Press, New York. (2) S. E. Koch and M. DesJardins and P. J. Kocin, 1983. ``An interactive Barnes objective map anlaysis scheme for use with satellite and conventional data,'', J. Climate Appl. Met., vol 22, p. 1487-1503.

The Barnes algorithm is as follows:

The gridded field is estimated iteratively. Successive iterations retain largescale features from previous iterations, while adding details at smaller scales.

The first estimate of the gridded field, here denoted `G_(ij)^0' (the superscript indicating the order of the iteration) is given by a weighted sum of the input data, with `z_k' denoting the k-th `z' value.

sum_1^n W_(ijk)^0 z_k G_(ij)^(0) = ---------------------- sum_1^n W_(ijk)0

where the notation `sum_1^n' means to sum the elements for the `k' index ranging from 1 to `n'.

The weights `W_(ijk)^0' are defined in terms of a Guassian function decaying with distance from observation point to grid point:

( (x_k - X_i)^2 (y_k - Y_j)^2 ) W_(ijk)^0 = exp(- -------------- - --------------- ) ( L_x^2 L_y^2 )

Here `L_x' and `L_y' are lengths which define the smallest `(x,y)' scales over which the gridded field will have significant variations (for details of the spectral response see Koch et al. 1983).

Note: if the user has supplied weights then these are applied in addition to the distance-based weights. That is, `w_i W_(ijk)' is used instead of `W_(ijk)'.

The second iteration derives a grid `G_(ij)^1' in terms of the first grid `G_(ij)^0' and ``analysis values'' `f_k^0' calculated at the `(x_k,y_k)' using a formula analogous to that above. (Interpolation based on the first estimate of the grid `G_(ij)^0' can also be used to calculate `f_k^0', with equivalent results for a grid of sufficiently fine mesh.) In this iteration, however, the weighted average is based on the difference between the data and the gridded field, so that no further adjustment of the gridded field is done in regions where it is already close to through the observed values. The second estimate of the gridded field is given by

sum_1^n W_(ijk)^1 (f_k - f_k^0) G_(ij)^1 = G_(ij)^0 + ------------------------------- sum_1^n W_(ijk)^1

where the weights `w_{ik,1}' are defined by analogy with `W_{ik}^0' except that `L_x' and `L_y' are replaced by `gamma^{1/2}L_x' and `gamma^{1/2}L_y'. The nondimensional parameter `gamma' (`0<gamma<1') controls the degree to which the focus is improved on the second iteration. Just as the weighting function forced the gridded field to be smooth over scales smaller than `L_x' and `L_y' on the first iteration, so it forces the second estimate of the gridded field to be smooth over the smaller scales `gamma^{1/2}L_x' and `gamma^{1/2}L_y'.

The first iteration yields a gridded field which represents the observations over scales larger than `(L_x,L_y)', while successive iterations fill in details at smaller scales, without greatly modifying the larger scale field.

In principle, the iterative process may be continued an arbitrary number of times, each time reducing the scale of variation in the gridded field by the factor `gamma^{1/2}'. Koch et al. 1983 suggest that there is little utility in performing more than two iterations, providing an appropriate choice of the focussing parameter `gamma' has been made. Thus the gridding procedure defines a gridded field based on three tunable parameters: `(L_x,L_y,gamma)'.

9.3.4.2: `convert columns to spline'

`convert columns to spline \
    [.gamma.] \
    [.xmin. .xmax. .xinc.]'

Fit a normal or taut interpolating spline, y=y(x), through the (x,y) data. Then subsample this spline to get a new set of (x,y) data. If the spline x-values, `.xmin.', etc, are not specified, the spline ranges from the smallest x-value with legitimate data to the largest one, with 200 steps in between.

The parameter `.gamma.' determines the type of spline used. If `.gamma.' is not specified, or is given as zero, a standard interpolating spline is used. A knot appears at each x location, with cubic polynomials spanning the space between the knots. If `.gamma.' lies between 0 and 6, a taut spline is used; this tends to have fewer wiggles than a normal spline. If `.gamma.' lies in the range 0 to 3, a taut spline is used, with the possible insertion of knots between interior x pairs. The value 2.5 is used commonly. If `.gamma.' lies in the range 3 to 6, extra knots are permitted in the x pairs at the ends of the domain. A value of 5.5 is used commonly.

Reference Chapter 16 of Carl de Boar, 1987. ``A Practical Guide to Splines'' Springer-Verlag.

read columns x y # function is y=x^2 0 0 1 1 2 4 3 9 4 16 set symbol size 0.2 draw symbol bullet convert columns to spline draw curve

9.3.4.3: `convert grid to columns'

`convert grid to columns'

Create column data from grid data. Each non-missing gridpoint is translated into a single (x,y,z) triplet. If column data already exist, then they are first erased. This command is useful in changing the grid configuration, perhaps from a non-uniform grid to a uniform grid. In the following example, a new grid with x=(0, 0.05, 0.1, ..., 0.1) and y=(10, 11, ..., 20) is created. The default gridding method (`convert columns to grid') is used here, but of course one is free to adjust the method as usual.

# ... read/create grid convert grid to columns delete grid set x grid 0 1 0.05 set y grid 10 20 1 convert columns to grid

9.3.4.4: `convert grid to image'

`convert grid to image [size .width. .height.] \
    [box .xleft. .ybottom. .xright. .ytop.]'

With no options specified, convert grid to a 128x128 image, using an image range as previously set by `set image range'.

Interpolation method: The interpolation scheme is the same used for contouring. Image points that lie outside the grid domain are considered missing. For points within the grid, the first step is to locate the patch of the grid upon which the pixel lies. Then the 4 neighboring grid points are examined, and one of the following cases holds.

1. If 3 or more of them are missing, the pixel is considered missing.
2. If just one of the neighboring grid points is missing, then the image pixel value is determined by bilinear interpolation on the remaining 3 non-missing grid points. (This amounts to fitting a plane to three measurements of height.)
3. If all 4 of the grid points are non-missing, then the rectangle defined by the grid patch is subdivided into four triangles. The triangles are defined by the two diagonal lines joining opposite corners of the rectangle. An ``image point'' is constructed at the center of the grid patch, with f(x,y) value taken to be the average of the values of the four neighbors. This value is taken to be the value at the common vertex of the four triangles, and then bilinear interpolation is used to calculate the image pixel value.

With the `size' options `.width.' and `.height.' specified, they set the number of rectanglular patches in the image.

With the `box' options specified, they set the bounding box for the image. If `box' is not given, the image spans the same bounding box as the grid as set by `set x grid' and `set y grid'.

Normally, missing values in the grid become white in the image, but this can be changed using the `set image missing value color to'... command.

9.3.4.5: `convert image to grid'

`convert image to grid'

Convert image to grid, using current graylevel/colorlevel mapping. For example, if one had a linear mapping of pixel values 0->255 into the user values 10->20, as in

set image range 10 20 set image grayscale black 10 white 20

then the output grid will be of value 10 where the pixel value is 0, etc. If the image is in color, the grid values will represent the result of mapping the colors to grayscale in the standard way (Foley and VanDam, 1984). [BUG: as of 1.063, the colorscale is ignored completely, and I'm not sure what happens.] The image data are interpolated onto the grid using a nearest-neighbor substitution. This command insists that the image x/y grids have already been defined.