9.3.28: `open'

• Opening Simple Files:
• Opening Pipes:
• Opening URLs:

9.3.28.1: Opening simple files

• Ascii Files:
• Binary Files:
• NetCDF Files:

open foo.dat

and then you can read the data using various commands. Thus a complete program might be

open foo.dat read columns x y draw curve

If a filename contains blanks or punctuation symbols, you must put it in double quotes (`"'), e.g.

open "foo bar.dat"

Indeed, Gri accepts double-quotes on any `open' command and some folks use it on all commands, as a matter of habit.

Gri can handle compressed files appropriately, e.g.

open foo.data.gz

so that there is no need to uncompress data for use with Gri.

Gri is quite persistant in looking for your file, and if a given file is not found, it will then check to see if a compressed version is available, and use that instead. Thus

open foo.dat

will look for a file named `foo.dat.gz' if `foo.dat' is not available. (Only files compressed with the GNU `gzip' utility are handled.)

If the `open' command was successful in opening the file, it will set the value of the synonym `\.return_value.' to the full pathname of the file. Thus, if `open a.dat' is done in directory `/home/gri', then `\.return_value.' will equal the string `/home/gri/a.dat'.

Binary Files Like most computer programs, Gri has some trouble with binary files. One big issue is the so-called "endian" character of the computer. Some computers store multi-byte values with the most significant bytes first, while others store them with the most significant bytes last. The problem is that nothing is stored in data files to indicate which convention was employed. For this reason, a version of Gri compiled on a so-called "big-endian" computer will misinterpret multi-byte values that were created on a so-called "little-endian" computer. Many folks in the scientific community have converted to using the NetCDF format (see NetCDF Files) for precisely this reason, since this format is independent of the endian character of the computer.

Presuming an appropriate endian character, however, reading is straightforward. A command of the form

open foo.dat binary

tells Gri that the data are stored in a binary format. With the above syntax, Gri expects images to be in `unsigned char' (8 bits), while other data, such as columns and grids, are expected to be in 32-bit format (suitable for reading into a so-called "float" variable in the C programming language).

You may also specify the format directly, as in the following examples; Gri then interprets all data as being in the indicated format and then converts to the internal format before using the data.

open \filename binary uchar open \filename binary 8bit open \filename binary int open \filename binary float open \filename binary double open \filename binary 16bit

As with ascii files, Gri will automatically uncompress any files that are compressed, and if it fails to find a given filename, it will try to open a compressed version of it (i.e. one with a `.gz' suffix).

NetCDF Files The NetCDF format provides the best of both worlds. It is binary, so that data are relatively compact, and may be read very quickly. (Reading ascii data is time-consuming in C++, the language in which Gri is written.) But it does not suffer the endian problem problem of normal binary files (see Binary Files), since information about the endian character is stored in the file itself, and Gri uses this information to decode the data without difficulty, regardless of the endian characteristics of the computer on which Gri is running and of the computer that created the data.

For more information on netCDF format, see

`http://www.unidata.ucar.edu/packages/netcdf/index.html' here .

The syntax of opening NetCDF files is as below

open foo.nc netCDF

and the syntax for reading such files is described in sections on the various `read' commands (see e.g. see Read Columns).

9.3.28.2: Opening pipes

Suppose we wish to plot an x-y plot using just the first few lines of a datafile named `foo.dat'. Unix users will know that a good way to see the first few lines of such a file would be to type the command `head foo.dat'. They also know that these lines could be provided to a second unix command, named `do_foo' say, by the command `head foo.dat | do_foo'. This uses a so-called "pipe", designated by the vertical line (called a pipe symbol below).

Gri can read the output from system commands by using a syntax in which the (quoted) system command ends in a pipe symbol, e.g.

open "head foo.dat |"

as in the example above.

Aside: When pipe-open commands are used, Gri creates a temporary file (often located in `/usr/tmp', but that varies with machine). This is automatically cleaned up when Gri completes executation, but if Gri dies (or is interrupted) before it finishes, you'll be left with an extra file in this temporary-storage directory. It's up to you to clean that directory up from time to time.

Some common examples of pipe-open commands are given below.

1. Comma-separated values are common in files created by, or intended for, spreadsheets. Since Gri expects data elements to be separated by blanks (or tabs), you'll have to convert the commas into blanks. There are many ways to do that using pipes, e.g. `sed' system utility, e.g.

   open "sed -e 's/,/ /g' foo.dat |"

   Other unix facilities, such as `tr' will also work, of course. If the file has headers, you'll want to remove them also. This can be done with the `skip' command (see Skip) but you could also do it at the open stage, e.g. to remove the first two lines, use

   open "sed -e 's/,/ /g' foo.dat | tail +2 |"

2. Manipulating column data is done by e.g.

   open "cat foo.dat | awk '{$1, $2 * 22}' |"

   where `awk' has been used to multiply the second column in the file named `foo.dat' by 22.

3. Time-based and geographical data are sometimes encountered. For an example, suppose that longitude/latitude (i.e. x/y) data are stored in Hour.minutesecond format, e.g. 12.2133 means hour 12, minute 21, second 33. Gri doesn't read HMS format, but gawk can be told to:

   open "cat datafile.HMS | \ awk '{ \ split($1, hms, \".\"); \ h = hms[1]; \ m = int(hms[2] / 100); \ s = hms[2] - 100 * m; \ x = h + m / 60 + s / 3600; \ split($2, hms, \".\"); \ h = hms[1]; \ m = int(hms[2] / 100); \ s = hms[2] - 100 * m; \ y = h + m / 60 + s / 3600; \ print(x,y) \ }' | " read columns x y

4. Timeseries data are often stored in formats that blend letters and numbers. For one thing, using letters (e.g. `aug') removes an ambiguity in numerically-based data. (Example: 02/03/2000 means one thing to an American and another thing in the rest of the world. However, everybody agrees on what 2000-Feb-03 means.) Suppose, for example, that we have data in a format such as

   Tue_Jul_25_11:07:51 0.62 Tue_Jul_25_11:22:51 0.59 Tue_Jul_25_11:37:51 0.56

   (stored in a file called `foo.dat' say) and we want a graph of the y-variable (0.62, 0.59, 0.56) versus x-variable, time expressed say as seconds in the day. Then here is how that could be done:

   open "cat foo.dat |\ sed -e 's/_/ /g' -e 's/:/ /g' |\ awk '{print ($4*3600+$5*60+$6, $7)}' |" read columns x y draw curve

   Note that the actual day information is skipped in this example; seasoned `awk' users could easily fill in the code to handle datasets spanning several days.

9.3.28.3: Opening URLs

The URL must be enclosed in quotes (since otherwise, Gri will interpret the `//' sequence as indicating an old way of denoting comments). For example,

open "http://gri.sourceforge.net/gridoc/examples/example1.dat" read columns x y show columns

If you don't have `wget' installed on your machine, the above won't work, but you can always use another fetching program, with a system call, as in the following:

\url = "http://gri.sourceforge.net/gridoc/html/examples/example1.dat" open "lynx -dump \url |" read columns x y draw curve