2 [1]The Columbia Crown The Kermit Project | Columbia University
3 612 West 115th Street, New York NY 10025 USA o [2]kermit@columbia.edu
5 [3]Home [4]Kermit 95 [5]C-Kermit [6]Scripts [7]Current [8]New [9]FAQ
8 [11]Table of platforms [12]Book: Using C-Kermit [13]Download
11 C-Kermit 9.0 Update Notes
14 * [15]How to Test Large-File Transfer
15 * [16]Arithmetic with Large Integers
16 * [17]FORCE-3 Packet Protocol
17 * [18]Variable Evaluation
19 * [19]The RENAME Command You Always Wanted
20 * [20]Other New Features
21 * [21]Incompatibilities
22 * [22]What's Not In C-Kermit 9.0
25 * [24]Demonstration: Secure POP mail fetcher
26 * [25]Demonstration: HP Switch Configuration Backup
27 * [26]Demonstration: HP iLO Blade Configuration
28 * [27]Demonstration: IBM/Rolm/Siemens CBX Management
29 * [28]Demonstration: CSV and TSV Files
30 * [29]Demonstration Scripts for Webmasters
32 This is the third supplement to [30]Using C-Kermit, Second Edition. I
33 apologize for the scattered nature of the information and I hope I can
34 organize it and gather it all into one place for easy and definitive
35 reference some day. It's a big job and it depends on the demand. For
36 the time being the definitive reference and introduction is the book
37 (which is now available also in a [31]Kindle Edition), plus the
38 [32]C-Kermit 7.0 update, [33]C-Kermit 8.0 update, and now this one.
39 Plus tons of other web pages on this site, sample script programs, and
42 In version 6.0, C-Kermit was a pretty powerful and flexible
43 communication program with scripting capabilities. By version 9.0, I'd
44 like to think of it more as a scripting language with built-in
45 communications. You can get an idea of the kinds of programs you can
46 write in Kermit language [34]here. You can develop programs quickly
47 because it's an interactive program, not a compiler. The scripting
48 language is the command language. Kind of like the Unix shell but
49 "somewhat" less cryptic, including concepts not only from C but from
50 PL/I, Snobol, LISP, and Smalltalk. The language itself is built upon
51 the command language of the much-loved [35]DECSYSTEM-20 from the 1970s
52 and 80s, the Clipper Ship of the Text Era. (Text is not a bad word.
53 Those of us who can touch-type and who are proficient in text-based
54 computing environments like Unix shell or VMS DCL are likely to be
55 orders of magnitude more productive than users of GUIs.)
57 - Frank da Cruz [36]fdc@columbia.edu
61 Very briefly, the major items:
62 * [37]Open Source license.
63 * [38]64-bit file access and transfer and 64-bit integer arithmetic
64 on most common platforms.
65 * Support for recent releases of Linux, Mac OS X, *BSD, etc ([39]see
67 * Support for newer OpenSSL releases up to and including 1.0.0d
69 * [41]Strengthened error checking for file transfer under extremely
71 * [42]Simplified semantics for variables used in scripts.
72 * Super-handy and useful [43]extensions to the RENAME command.
73 * Many other scripting improvements including support for reading and
74 writing [44]CSV and TSV files.
75 * [45]MIME character-set names are now recognized.
76 * Improved logging and debugging (see demo [46]here).
77 * Lots more described or listed below, and [47]here.
81 C-Kermit 9.0 has the [48]Revised 3-Clause BSD License, an open source
82 license approved by OSI, the [49]Open Source Initiative.
86 Kermit is, first and foremost, a file-transfer program. One might
87 expect it to be able to transfer any kind of file, but that has been
88 decreasingly the case as file sizes began to cross the 2 gigabyte
91 The biggest change since C-Kermit 8.0.211 is support for large files on
92 platforms that support them. A "large file" is one whose size is
93 greater than 2^31-1 (2,147,483,647) bytes (2GB-1); that is, one whose
94 size requires more than 31 bits to represent. Before now, Kermit was
95 able to access such files only on 100% 64-bit platforms such as Digital
96 Unix, later known as Tru64 Unix. In the new release, Kermit takes
97 advantage of the X/Open Single UNIX Specification Version 2 (UNIX 98)
98 Large File Support (LFS) specification, which allows 32-bit platforms
99 to create, access, and manage files larger than 2GB.
101 Accommodating large files required code changes in many modules,
102 affecting not only file transfer, but also file management functions
103 from directory listings to local file manipulation, plus the user
104 interface itself to allow entry and display of large numbers. All this
105 had to be done in a way that would not affect pure 32-bit builds on
106 platforms that do not support large files. Large file support is
107 summarized in the [50]Table of Platforms; entries in Yellow (32-bit
108 builds that support 64-bit integers) and Green (64-bit builds) support
111 Note that VMS C-Kermit and Kermit 95 for Windows have always been able
112 to transfer large files. However their user interface used 32-bit
113 integers for statistics and the file transfer display. In C-Kermit 9.0
114 Alpha.03, VMS C-Kermit on 64-bit platforms (Alpha and Itanium) should
115 now give correct statistics and progress displays. (We'll see about
118 How to Test Large-File Transfer
120 Several methods are available for testing large-file transfers:
121 * By transferring a real file that is more than 2147483648 bytes long
122 (a file whose length requires more than 31 bits to express); or to
123 be totally sure, that is longer than 4294967296 bytes (32 bits or
124 more). Or to be double super sure, longer than 8589934592 (33
126 * If you don't have such a file or there is not sufficient disk space
127 for such a file, you can create a special kind of file that takes
128 up one block on the disk but appears to be 4.3GB long by compiling
129 and running [51]THIS C PROGRAM on Linux, Solaris, HP-UX, or other
130 Unix platform that supports large files. Kermit or FTP or any other
131 file transfer program will transfer the result (BIGFILE) in such a
132 way as to actually put 4.3GB (or other desired size; see source) on
134 * You can use Kermit's CALIBRATE feature to transfer a large file
135 that doesn't exist. At the receiver, use RECEIVE /CALIBRATE. At the
136 sender, use SEND /CALIBRATE:length, e.g.:
138 (At remote kermit...)
140 C-Kermit> receive /calibrate
141 (Return to local kermit...)
143 C-Kermit> send /calibrate:4300000000
144 This sends a simulated file 4.3GB in length, that does not exist on
145 the sender and will not take up any disk space on the receiver.
146 SEND /CALIBRATE: accepts big numbers only in Kermit versions that
147 support them (this does not include Kermit 95 on Windows). This
148 method tests only Kermit's ability to express and understand large
149 file sizes, but does not test Kermit's file-system interface, since
150 no files are involved.
152 Arithmetic with Large Integers
154 Because large file support requires the availability of a 64-bit signed
155 integer data type, other aspects of C-Kermit were adapted to use it
156 too, most notably Kermit's algebraic expression evaluator and its
157 [52]S-Expression interpreter, on all platforms that support large files
158 (those listed as 64 or 32/64 in the Word column of the [53]table). In
159 fact, every Kermit command that parses a number in any field can now
160 parse a large number on those platforms.
162 S-Expressions can now be forced to operate with integers only, without
163 floating-point conversion or having to explicitly truncate each result;
164 as an example. see the revised [54]Easter date calculation script.
166 FORCE-3 Packet Protocol
168 The Kermit protocol has proven itself over the past 30 years to be
169 robust in terms of surviving harsh transmission environments and
170 delivering the data correctly and completely. In these times of
171 Internet everywhere and error-correcting modems in the few places where
172 the Internet isn't, few people even recall the kinds of difficult
173 conditions that were common when the Kermit protocol was first
174 developed: noisy telephone lines, serial interfaces that drop
175 characters, lack of transparency to control or 8-bit characters,
176 absence of flow control, "bare" modems without error correction.
178 But the Internet is not everywhere, and not all modems are
179 error-correcting. Perhaps the most difficult trial so far for Kermit or
180 any other protocol is the [55]EM-APEX project, in which floats are
181 dropped into the ocean from an aircraft into the path of a hurricane;
182 these floats dive into the water measuring current, temperature, and
183 salinity at different depths and then surfacing to phone home, sending
184 the data to land stations using Kermit protocol over
185 non-error-correcting 300bps [56]Iridium satellite modems, with high
186 seas and winds battering the floats and heavy ([57]sometimes
187 electrical) storms between the modem and the satellite.
189 Because of the transmission speed and long distances involved, the
190 transfers were very slow. The Kermit software in the floats is
191 [58]Embedded Kermit, which did not implement sliding windows, which
192 would have sped up the flow considerably. John Dunlap, engineer at the
193 University of Washington's Applied Physics Laboratory, undertook the
194 task of adding sliding windows to E-Kermit. For testing, he rigged up a
195 [59]simulator in which Kermit transfers take place over a connection
196 with different amounts of noise and delay. He found that occasionally,
197 a transfer would appear to succeed, but the received file would be
200 According to the Kermit protocol definition, the first packet always
201 has block-check type 1, a 6-bit checksum, which is the only block check
202 type that all Kermit implementations are required to support; thus any
203 Kermit partner can process this packet. This packet itself can
204 negotiate a higher level of checking, such that subsequent packets have
205 (say) block-check type 3, a 16-bit cyclic redundancy check (CRC)
206 encoded as three printable 7-bit ASCII characters. The 16-bit CRC can
207 catch all errors of certain kinds (single-bit, double-bit, bursts of 16
208 bits or less), and more than 99.9984741210937% of all other possible
211 John's simulations revealed that file corruption could occur undetected
212 when the initial packet was corrupted in such a way that a parameter or
213 capability byte was changed and the checksum also changed to make the
214 packet appear to be correct, thus allowing the transfer to proceed with
215 the two Kermit partners out of sync as to packet encoding and
216 interpretation (the chances of two such errors producing a seemingly
217 valid packet are about 1 in 6000 when using the 6-bit checksum). For
218 example the compression technique might be misnegotiated and then the
219 receiver might store incoming data without decompressing it.
221 The solution is a new option, selected by:
225 to require a type 3 block check (16-bit CRC) on every packet, including
226 the initial ones, thus reducing the probability of a misnegotiation by
227 many orders of magnitude. THIS PARAMETER CAN NOT BE NEGOTIATED. Each
228 Kermit program must be given the "set block 5" command prior to
229 transfer. That's because normally every Kermit program expects the
230 first packet to have a 6-bit checksum, and if the first packet has a
231 3-byte, 16-bit CRC, the packet receiver will think it is corrupted.
233 In practice, however, it is possible to code the packet receiver
234 "cheat" by reading the packet data before verifying the block check.
235 Thus when the receiver is C-Kermit 9.0 Beta.01 or later or E-Kermit 1.7
236 or later, it is only necessary to give the "set block 5" command to the
237 file sender, and the receiver will check for a FORCE-3 first packet. If
238 the receiver does not support this feature, however, the the initial
239 packet will be be rejected (after several retries) and the file
240 transfer will not take place. There is no attempt to "back off" to
243 CAPTION: Table 4. Kermit Protocol Packet Block Check Types
245 Type Command Bytes Status Explanation
246 1 SET BLOCK 1 1 Required in all Kermit implementations. Negotiated.
247 6-bit checksum, suitable for good connections.
248 2 SET BLOCK 2 2 Optional, negotiated. 12-bit checksum. 64 times
249 stronger than type 1.
250 3 SET BLOCK 3 3 Optional, negotiated. 16-bit CRC.
251 BLANK-FREE-2 SET BLOCK 4 2 Optional, negotiated. 12-bit checksum, two
253 FORCE-3 SET BLOCK 5 3 Optional, not negotiated. 16-bit CRC forced all
260 Does the strange behavior of Kermit's \%x variables puzzle or annoy
263 Kermit software development has been a collaborative project over the
264 years, with contributions coming in from almost every country and every
265 sector of the economy - academic, corporate, government. Thus not all
266 versions, and not all features of a given version, are a product of
269 One example was the introduction of variables for text substitution,
270 first in a version of MS-DOS Kermit that was sent in by someone
271 somewhere (I could look it up, but no time...) Although the design of
272 the notation for variable names (table below) is mine, the underlying
273 code was contributed. In that code there was only one kind of variable,
274 and if I recall correctly the variable name was a backslash followed by
275 a single letter, for example \a, \b, etc. The contributed code
276 evaluated these variables recursively, meaning if the definition of a
277 variable contained variable references, then these were resolved when
278 dererencing the variable, and the process would continue as deep down
279 as necessary to resolve the thing fully.
281 This was sometimes handy, but it had one severe drawback: There was no
282 way to use variables in a straightforward way to represent strings that
283 contained literal backslashes; for example, DOS or Windows pathnames.
284 This gave rise to all kinds of quoting rules and conventions (e.g.
285 doubling backslashes or forcing single-level evaluation with
286 \\fcontents()), and also to the introduction of other kinds of
287 variables that were evaluated one level deep, rather than recursively.
289 To accommodate coexistence of different kinds of variables as well as
290 "escape sequences" for representing control and 8-bit characters, the
291 syntax for variable names was extended to include three elements: the
292 leading backslash, then a single character indicating the type of
293 variable, and then the name of the variable in a format corresponding
294 to the type designator, as shown in this somewhat simplified table:
296 CAPTION: Table 1. Variable-name Syntax in Kermit
299 \000 - \255 8-bit character constant (decimal)
300 \d000 - \d255 Alternative notation for 8-bit character (byte) constant
302 \o000 - \o377 8-bit character constant (octal)
303 \x00 - \xff 8-bit character constant (hexadecimal)
304 \%a - \%z Scalar variable, evaluated recursively.
305 \%0 - \%9 Macro argument, scalar, evaluated recursively.
307 \&a[x] Array reference, evaluated recursively (x is any constant or
309 \v(name) Built-in scalar variable, evaluated one level deep.
310 \m(name) User-defined scalar variable, evaluated one level deep.
311 \$(name) An environment variable, evaluated one level deep.
312 \s(name[n:m]) Compact substring notation, evaluated one level deep.
313 \fname(args...)) Built-in function with zero or more arguments.
315 \N OUTPUT comand only: NUL, ASCII 0
316 \B OUTPUT comand only: BREAK
317 \L OUTPUT comand only: Long BREAK
319 Variable names in Kermit are case-independent. The simplifications in
320 the table are that the notation for decimal and octal bytes can have
321 from one to three digits, and can include braces to separate them from
322 text digits, e.g. \7, \{123}, \o{50}. Hex bytes too, except they must
323 always have exactly two hex digits, 0-9a-f. Array indices must be, or
324 must evaluate to, numbers (floating point numbers are truncated).
325 Associative arrays are also available (dynamic arrays with arbitrary
326 text as subscript), but they are really just a variation on \m()
327 variables (read about associative arrays [61]here). Also, there are
328 some alternative notations for compact substring notation.
330 We didn't want to have lots of "distinguished" characters, as the UNIX
331 shell does; one is enough, clarity over brevity. Although the notation
332 can be a bit cumbersome, we can use the \m(name) form to circumvent the
333 overevaluation in most contexts. But macro arguments are always
334 assigned to the \%0-9 variables, and thus always evaluated recursively,
335 making it difficult and confusing to pass (e.g.) Windows pathnames as
336 arguments to macros. The same is true for array elements, especially in
337 contexts where they are used to return results from built-in functions
338 (for example, \fsplit() used to return the elements of a
339 [62]comma-separated value list if any of the values contained
340 backslashes). An even worse scenario is when macro arguments are passed
341 from one macro to another; for some graphic illustrations see
342 [63]Taming the Wild Backslash - Part Deux from the [64]C-Kermit 7.0
345 We can't just change how variables are evaluated because that would
346 break existing scripts. But we can always add Yet Another SET Command:
348 SET COMMAND VARIABLE-EVALUATION { RECURSIVE, SIMPLE }
350 This applies only to \%a-z and \%0-9 variables and to \&a-z[] arrays
351 (since all other kinds of variables are evaluated only one level deep).
352 The default, of course, for backwards compatibility, is RECURSIVE.
353 SIMPLE forces the evaluation of these variables to return their literal
354 contents, without further evaluation:
356 * An exception is made in the case of array subscripts, because
357 changing how they are evaluated could break a lot of scripts, and
358 anyway there should never be any harm in evaluating them
359 recursively because their final value is always (or should be)
360 numeric, not some string that might contain backslashes.
361 * The VARIABLE-EVALUTION setting is on the command stack. Thus you
362 can give this command in a macro, command file, or user-defined
363 function without affecting the calling environment.
364 * The new \frecurse() function forces recursive evaluation of its
365 argument regardless of the VARIABLE-EVALUATION setting. The
366 argument can be any string (or nothing at all); all the variables
367 in the string, even \m() ones, are evaluated recursively:
373 echo \frecurse(\m(xx))
375 echo \frecurse(it's as easy as \m(xx))
376 it's as easy as easy as 1 2 3
378 * The new \v(vareval) built-in variable contains the current setting
379 (recursive or simple) at the current command-stack level.
381 Here's a short script for illustration:
383 define path c:\users\fdc\somefile.txt
384 define test1 { # Normal recursive argument evaluation
387 define test2 { # Simple argument evaluation
395 And here's the result:
397 ?<ERROR:NO_SUCH_FUNCTION:\fdc\somefile.txt()>
398 test2: arg=c:\users\fdc\somefile.txt
400 The first line might seem surprising, but under the normal rules (see
401 table above) \f indicates a function call, with the letters following
402 the 'f' being the name of the function. But there is no function by
403 that name... and if there were, you probably didn't intend to call it!
405 SET COMMAND VARIABLE-EVALUATION SIMPLE has no effect on constants, only
406 on variables. Note how \m(path) is defined. The DEFINE command assigns
407 the literal value of its argument to the named variable (see Table 3
408 below), thus in this case no special syntax is needed. But in other
409 contexts, you must double the backslashes or use the \fliteral()
410 function to use literal backslashes in data:
412 test2 c:\\users\\fdc\\somefile.txt
413 test2 \fliteral(c:\users\fdc\somefile.txt)
415 C-Kermit 9.0 adds a new notation for \fliteral() which also has certain
416 advantages over it: \q(string)
419 test2 \q(c:\users\fdc\somefile.txt)
421 Since \fliteral() is a function, its argument list (the text within
422 parantheses) has special syntax of its own, in which commas and braces
423 are treated specially and introduce another set of quoting problems.
424 \q(string) doesn't have these problems. The only consideration is that
425 parentheses must be balanced or else quoted (preceded by backslash), or
426 represented as numeric character entities (left paren = \40, (right
429 Or else hold the value in a simple variable as we did with \\m(path)
432 SET COMMAND VARIABLE-EVALUATION SIMPLE is a big change and might have
433 repurcussions that didn't show up in the initial tests; a lot more
436 On the topic of variables, let's summarize in one place the ways in
437 which values can be explicitly assigned to variables. There is nothing
438 new here except the table itself:
440 CAPTION: Table 2. Variable Assignment in Kermit
442 Command Shorthand Explanation
443 DEFINE name value .name = value The literal value becomes the contents
444 of the named variable; variables names in the value are copied without
445 evaluation. This command is for defining macros that take parameters,
446 as well as for defining simple variables, especially if the values
448 _DEFINE name value Like DEFINE but the name is evaluated before use.
449 ASSIGN name value .name := value The value is evaluated and the result
450 becomes the contents of the named variable.
451 _ASSIGN name value Like ASSIGN but the name is evaluated before use.
452 EVALUATE name expression .name ::= value The expression (in regular
453 algebraic notation) is evaluated arithmetically and the result becomes
454 the contents of the named variable. If the expression contains any
455 variables they are evaluated first.
456 _EVALUATE name expression Like EVALUATE but the name is evaluated
458 INCREMENT name expression Evaluates the variables in the expression,
459 then evaluates the expression arithmetically, and then adds the value
460 to the contents of the named variable, which must be a number or an
461 algebraic expression. If the expression is empty, a value of 1 is used.
462 _INCREMENT name expression Like INCREMENT but the name is evaluated
464 DECREMENT name expression Evaluates the variables in the expression,
465 then evaluates the expression arithmetically, and then subtracts the
466 value from the contents of the named variable, which must be a number
467 or an algebraic expression. If the expression is empty, a value of 1 is
469 _DECREMENT name expression Like DECREMENT but the name is evaluated
471 DECLARE name = list An array declaration can include an initializer
472 list; items in the list are evaluated before assignment. This can be
473 defeated by doubling any backslashes or enclosing individual arguments
475 DO name arguments name arguments When invoking a macro with a DO
476 command (or an implied one), the arguments are evaluated, then assigned
477 to \%1, \%2, etc, and the macro's name to \%0.
478 (SETQ name value) Kermit also includes a mini-[65]LISP intpreter
480 Variables are evaluated automatically in Kermit commands simply by
481 referencing them, according to rules given in Table 1. The following
482 functions can be used to change how a a particular variable is
485 CAPTION: Table 3. Kermit Functions for Evaluating Variables
487 Function Argument Description
488 \fcontents() \%x or \&x[y] Evaluates the variable or array element
489 (which normally would be evaluated recursively) one level deep.
490 \fdefinition() name If the argument is a \%x variable or an array
491 element, it is evaluated to get the name; otherwise the argument is the
492 name. Its definition is returned with no recursion.
493 \m() name Equivalent to \fdefinition().
494 \recurse() \m(name) Forces recursive evaluation of a macro definition
495 (a.k.a. long variable name). NOTE: \frecurse() can operate on any kind
496 of variable as well as on any string containing any mixture of
499 C-Kermit's RENAME Command
501 C-Kermit's RENAME command, which is used for changing the names of
502 local files or for moving files locally, has two basic forms:
504 RENAME [ optional-switches ] oldfilename newfilename
505 This form lets you change the name of a single file from
506 oldfilename to newfilename. Example:
507 rename thismonth.log lastmonth.log
509 RENAME [ optional-switches ] filespec directoryname
510 This form lets you move (without renaming) one or more files
511 (all the files that match the filespec, which may contain
512 wildcard characters such as "*") to the given directory.
514 rename *.txt ~/textfiles/
516 Traditionally, the optional switches have been:
518 RENAME /LIST oldname newname
519 Display the old and new name for each file while renaming.
520 Synonyms: /LOG, /VERBOSE. Example:
521 rename /list *.txt ~/textfiles/
523 RENAME /NOLIST oldname newname
524 Don't display the old and new name for each file while renaming.
525 This is the default behavior. Synonyms: /NOLOG, /QUIET. Example:
526 rename /nolist *.txt ~/textfiles/
528 Reminder: Every switch starts with a slash (/) and must be preceded by
531 New RENAME Features for C-Kermit 9.0
533 A series of new options (switches) have been added to let you change
534 the names of multiple files at once by case conversion, string
535 substitution, or character-set conversion, and optionally also move
536 them to a different directory:
538 /LOWER: Convert the filename to lowercase
539 /UPPER: Convert the filename to uppercase
540 /CONVERT: Change the filename's character encoding
541 /REPLACE: Do string substitutions on the filename
543 If the source-file specification includes a path or directory, any
544 changes are applied to the filenames only, not to the directory or path
547 Since name changes, when applied to many files at once, can have
548 consequences that are not easily undone, there are also some new
549 controls, safeguards, and conveniences:
552 This switch tells Kermit to show you what the RENAME command
553 would do without actually doing it. /SIMULATE implies /LIST.
555 RENAME /COLLISION:{FAIL,SKIP,OVERWRITE}
556 This switch governs Kermit's behavior when renaming multiple
557 files, and any of the names would collide with the name of a
558 file that already exists. The default, for compatibility with
559 earlier releases of C-Kermit, is OVERWRITE, i.e. write over the
560 existing file. The other two protect existing files. SKIP means
561 to skip (not rename) the file that would cause the collision,
562 and proceed to the next file, if any. FAIL means that no files
563 will be renamed if there would be any collisions; for this
564 Kermit makes two passes, checking each new name it constructs
565 for existence before starting the second pass (however, there is
566 no guarantee that in the second pass, it won't create the same
567 new name for more than one file; in that case, it will stop
568 before executing the second rename). Example:
569 rename /simulate /collision:proceed * ~/tmp/
571 Reminder: In switches such as /COLLISION that take arguments
572 (operands), the switch name and its argument(s) are separated by a
573 colon (:) with no intevening spaces. Also remember that Kermit keywords
574 can always be abbreviated by leaving off characters from the right, as
575 long as the result is still unique in its context. Thus "ren /col:f"
576 would be equivalent to "rename /collision:fail".
578 You can change the following preferences for the RENAME command with
579 the new SET RENAME command:
581 SET RENAME LIST { ON, OFF }
582 Tells the RENAME command whether to list its actions if you
583 don't include a /LIST or /NOLIST or equivalent switch.
585 SET RENAME COLLISION { FAIL, OVERWRITE, SKIP }
586 Tells the RENAME command how to handle filename collisions in
587 the absence of a /COLLISION switch. That is, it replaces the
588 default action of OVERWRITE with action of your choosing, which
589 is then used in any RENAME command that does not include an
590 explicit /COLLISION switch.
593 Displays the current SET RENAME settings.
595 Changing the Case of Filenames
597 RENAME /UPPER:{ALL,LOWER} filespec [ directory ]
598 RENAME /LOWER:{ALL,UPPER} filespec [ directory ]
599 These switches let you change the alphabetic case of letters in
600 all the files whose names match the filespec. If a directory
601 name is given after the filespec, then the files are also moved
602 to the given directory.
604 By default, all files that match the given filespec have their names
605 changed (if necessary). This is what the ALL argument means, e.g.:
610 You can use either form: RENAME /LOWER is equivalent to RENAME
611 /LOWER:ALL. The other argument (/LOWER:UPPER or /UPPER:LOWER) means to
612 leave mixed-case filenames alone, and rename only those files whose
613 names contain letters of only the given case. Examples:
615 RENAME /UPPER:ALL foo.bar
616 Changes the filename to FOO.BAR.
618 RENAME /UPPER foo.bar
619 Same as "rename /upper:all foo.bar".
621 RENAME /UPPER foo.bar ~/old/
622 Renames foo.bar to FOO.BAR and moves it to the user's old
626 Changes the names of all files to have only lowercase letters.
628 RENAME /LOWER:UPPER *
629 Changes the names of only those files whose names contain no
630 lowercase letters to have only lowercase letters. For example,
631 FOO.BAR would be changed, Foo.Bar would not be changed. foo.bar
632 would not be changed either because it's already all lowercase.
634 RENAME /LOWER:UPPER * ~/new/
635 Same as the previous example, but also moves each file to the
636 user's new directory (whether it was renamed or not).
638 Case conversion works reliably for ASCII characters only. Kermit uses
639 the C library for this, which on any given platform might or might not
640 handle non-ASCII letters, and if it does, then how it works would
641 normally depend on your locale definitions (the LC_CTYPE and/or LANG
642 environment variable in Unix). When non-ASCII letters are not handled
643 by the C library, the RENAME command does change their case. For
644 example, Olga_Tañón.txt might become OLGA_TAñóN.TXT.
646 String Replacement in Filenames
648 The RENAME command also lets you change filenames by string
651 RENAME /FIXSPACES[:String] filespec [ directory ]
652 Replaces all spaces in each matching filename by the given
653 string, if any, or if none is given, by underscore. Examples:
656 RENAME /FIXSPACES:_ *
657 RENAME /FIXSPACES:"" *
658 RENAME /FIXSPACES:<040> *
660 The first two are equivalent, replacing each space with
661 underscore; a file called "My Favorite Photo.jpg" becomes
662 "My_Favorite_Photo.jpg". The third example removes all spaces
663 ("MyFavoritePhoto.jpg"). The fourth replaces each space with the
664 string "<040>" ("My<040>Favorite<040>Photo.jpg").
666 RENAME /REPLACE:{{String1}{String2}} filespec [ directory ]
667 Renames each matching file by changing occurrences of String1 in
668 its name to String2. If a directory specification is included,
669 the file is also moved to the given directory (even if the name
670 was not changed). Note that in this case, the curly braces are
671 part of the command. Example:
673 RENAME /REPLACE:{{.jpeg}{.jpg}} *
675 changes all *.jpeg files to *.jpg.
677 By default, RENAME /REPLACE changes all occurrences of String1 in each
678 filename to String2 so, for example, if you had a file called
679 abcjpegxyz.jpeg, the command just shown would change its name to
682 For greater control and flexibility, the /REPLACE: switch argument can
683 take several distinct forms:
685 RENAME /REPLACE:String1 filespec [ directory ]
686 This means to remove all occurrences of String1 from the given
687 filenames name. It is equivalent to /REPLACE:{{String1}{}}. A
688 handy use for this option is to remove spaces from filenames.
690 RENAME /REPLACE:{{String1}{String2}} filespec [ directory ]
691 As already noted, this replaces every occurrence of String1 with
692 String2 in each filename. Alphabetic case in string matching is
693 done according to the current SET CASE setting.
695 RENAME /REPLACE:{{ }{_}} filespec [ directory ]
696 This replaces all spaces in the given filenames with underscore,
697 equivalent to RENAME /FIXSPACES.
699 RENAME /REPLACE:{{String1}{String2}{Options}} filespec [ directory ]
700 Options can be included that add more control to the process.
701 The option string is a sequence of characters; each character in
702 the string is an option. The choices are:
704 A String matching is to be case-sensitive, regardless of SET CASE.
705 a String matching is to be case-independent, regardless of SET CASE.
706 ^ String replacement will occur only at the beginning of the filename.
707 $ String replacement will occur only at the end of the filename.
708 1 Only the first occurrence of the string will be replaced.
709 2 Only the second occurrence of the string will be replaced.
711 9 Only the ninth occurrence of the string will be replaced.
712 - (hyphen, minus sign) Before a digit: occurrences will be counted from
714 ~ (tilde) Before digit or minus sign: all occurrences but the given one
717 The tilde modifier works only with single-byte character sets such as
718 ASCII, CP437, ISO 8859-1, etc, but not with multibyte character sets
719 such as UCS2, UTF8, or any of the Japanese Kanji sets.
721 Here are some examples showing how to use the /REPLACE options:
723 RENAME /REPLACE:{{foo}{bar}{^}} *
724 For all files whose names start with "foo", replaces the "foo"
725 at the beginning with "bar".
727 RENAME /REPLACE:{{}{New-}{^}} *
728 Prepends "New-" to the name of each file.
730 RENAME /REPLACE:{{.jpeg}{.jpg}{$}} *
731 Replaces ".jpeg" at the end of each filename with ".jpg".
733 RENAME /REPLACE:{{}{-Old}{$}} *
734 Appends "-Old" to the name of each file.
736 RENAME /REPLACE:{{foo}{bar}{a}} *
737 Replaces "foo", "FOO", "Foo", "fOO", etc, with "bar" in each
740 RENAME /REPLACE:{{foo}{bar}{A}} *
741 Replaces only (lowercase) "foo" in filenames with "bar".
743 RENAME /REPLACE:{{a}{XX}} *
744 Changes every "a" to "XX". For example a file called "a.a.a.a"
745 would become "XX.XX.XX.XX".
747 RENAME /REPLACE:{{a}{X}{2}}
748 Changes only the second "a" to "X". For example a file called
749 "a.a.a.a" would become "a.X.a.a".
751 RENAME /REPLACE:{{a}{X}{-1}}
752 Changes only the final "a" in the filename (it doesn't have to
753 be at the end) to "X". For example a file called "a.b.a.c.a.d"
754 would become "a.b.a.c.X.d".
756 RENAME /REPLACE:{{foo}{NOTFOO}{-2}}
757 Changes the second-to-last "foo" (if any) in the filename to
760 RENAME /REPLACE:{{foo}{}{-2}}
761 Deletes the second-to-last "foo" (if any) from the filename.
763 RENAME /REPLACE:{{.}{_}{~1}}
764 Changes all but the first period to an underscore; for example,
765 "a.b.c.d.e" would become "a.b_c_d_e".
767 RENAME /REPLACE:{{.}{_}{~-1}}
768 Changes all but the final period to an underscore; for example,
769 "a.b.c.d.e" would become "a_b_c_d.e".
771 In the Options field, digits (and their modifiers), ^, and $ are
772 mutually exclusive. If you include more than one of these in the option
773 string, only the last one is used. Similarly for 'a' and 'A':
775 RENAME /REPLACE:{{foo}{bar}{Aa2$^}} *
776 This replaces "foo" with "bar" no matter what combination of
777 upper and lower case letters are used in "foo" ('a' overrides
778 'A' in the option string), but only if "foo" is at the beginning
779 of the filename ('^' overrides '$' and '2').
781 If you give an /UPPER or /LOWER switch and a /REPLACE switch in the
782 same RENAME command, the /REPLACE action occurs first, then the case
785 RENAME /REPLACE:{{foo}{bar}} /UPPER * /tmp
786 For each file: changes all occurrences of "foo" in the name to
787 "bar", then converts the result to uppercase, and then moves the
788 file to the /tmp directory. So (for example) "foot.txt" would
789 become "/tmp/BART.TXT".
791 Changing the Character Encoding of Filenames
793 As you know, text is represented on the computer as a series of
794 numbers, with a given number corresponding to a given character
795 according to some convention or standard. Filenames are represented the
796 same way. The trouble is, different computers, or even different
797 applications on the same computer, might use different standards or
798 conventions ("character sets") for representing the same characters.
799 Usually ASCII is safe, but anything beyond that -- non-ASCII characters
800 such as accented or non-Roman letters -- is likely to vary. Sometimes
801 you have text that's in the "wrong" character set and you need to
802 convert it to something you can can use. Kermit has always been able to
803 handle this as part of file transfer and terminal emulation, as well as
804 being able to convert text files locally with its TRANSLATE command.
805 Now there's a way to convert filenames too, for example after copying
806 files from a CD that uses a different encoding:
808 RENAME /CONVERT:charset1:charset2 filespec [ directory ]
809 Converts filenames from the first character set to the second
810 one. The two character sets can be chosen from the SET FILE
811 CHARACTER-SET list; for complete details see [66]this page. For
812 example suppose you have a file called "Olga_Tañón.txt" on a
813 computer where ISO 8859-1 Latin Alphabet 1 is used, and you have
814 transported it (e.g. on CDROM) to another computer where the
815 text encoding is UTF8. Maybe you also have a lot of other files
816 with similar names in the same directory. You can convert the
817 filenames to UTF8 like this:
819 RENAME /CONVERT:latin1:utf8 *
821 /CONVERT can not be combined with /UPPER, /LOWER, or /REPLACE.
823 You should NOT use UCS2 for filenames since this encoding is not
824 compatible with C strings used in Unix and elsewhere.
826 RENAME /CONVERT affects only the filename, not the file's contents. You
827 can use the TRANSLATE command to convert the encoding of the contents
832 See the [67]C-Kermit Daily Builds page for details. Very briefly:
834 * Perhaps most important, modernized makefile targets for the major
835 Unix platforms: Linux, Mac OS X, AIX, Solaris, etc. These are
836 somewhat automated; not autoconf exactly, but they cut down
837 significantly on redundant targets. For example, one single "linux"
838 target works on many (hopefully all) different Linux
839 configurations, where before different targets were required for
840 different combinations of (e.g.) curses / ncurses / no curses;
841 32-bit / 64-bit; different feature sets and library locations.
842 (Separate targets are still required for Kerberos and/or SSL
843 builds, but they are "subroutinized".)
844 * Bigger buffers, more storage for commands, macros, scripts,
845 strings, and filename expansion in 64-bit versions and in 32-bit
846 versions that support large files.
847 * New options for the RENAME command, allowing you to rename groups
848 of files at once, changing case of letters in the name or changing
849 its character set, removing spaces or changing them to something
850 else, and/or doing anchored or floating or occurrence-based string
851 replacement, described [68]HERE.
852 * Built-in FTP client for VMS. This is the [69]same FTP client Unix
853 C-Kermit has had since version 8.0, minimally adapted to VMS by
854 SMS, supporting binary and Stream_LF file transfer only (in other
855 words, nothing to handle RMS files), but otherwise fully functional
856 (and scriptable) and theoretically capable of making connections
857 secured by SSL (at least it compiles and links OK with SSL - HP SSL
858 1.3 in this case). In the present Alpha release, this is an
859 optional feature requested by including the "i" option in P1 (and
860 by including "CK_SSL" in P3 if you also want SSL, and then also
861 "OPENSSL_DISABLE_OLD_DES_SUPPORT" if necessary). Much testing is
862 needed to determine if it should be included in the final C-Kermit
864 * Large file support in VMS, also by SMS. Alpha and Itanium only (not
865 VAX). VMS C-Kermit was already able to transfer large files, but
866 the file-transfer display (numbers and progress bar) and statistics
867 were wrong because they used ints. In the present Alpha test
868 release, this is an optional feature requested by including the "f"
870 * User-settable FTP timeout, works on both the data and control
872 * FTP access to ports higher than 16383.
873 * New PUTENV command that allows Kermit to pass environment variables
874 to subprocesses (Unix only).
875 * Unix C-Kermit SET TERMINAL TYPE now passes its arguments to
876 subprocesses as an environment variable.
877 * New TOUCH command, many file selection options.
878 * New DIRECTORY command options and switches (/TOP, /COUNT;
879 HDIRECTORY, WDIRECTORY...). To see the ten biggest files in the
880 current directory: "dir /top:10 /sort:size /reverse *" or
881 equivalently, "hdir /top:10 *". WDIR lists files in reverse
882 chronological order, shorthand for "dir /sort:date /reverse".
883 * New command FSEEK /FIND:string-or-pattern, seeks to the first line
884 in an FOPEN'd file that contains the given string or matching the
885 given pattern. Example: Suppose you have a file of lines like this:
887 quantity description...
888 in which the first "word" is a number, and a description (for
889 example, the name of an item). Here is how to use FSEEK to quickly
890 get the total quantity of any given item, which is passed as a
891 parameter (either a literal string or a pattern) on the command
894 #!/usr/local/bin/kermit +
895 if not def \%1 exit 1 Usage: \fbasename(\%0) string-or-pattern
897 .filename = /usr/local/data/items.log # Substitute the actual filename
898 set case off # Searches are case-independent
899 fopen /read \%c \m(filename) # Open the file
900 if fail exit 1 "\m(filename): \v(errstring)" # Fail: exit with error message
901 .total = 0 # OK: Initialize the total
902 echo Searching "\%1"...
905 fseek /line /relative /find:\%1 \%c 0 # Get next line that has target
906 if fail break # Failure indicates EOF
907 fread /line \%c line # Read it
908 if fail break # (shouldn't happen)
909 increment total \fword(\m(line),1) # Increment the total
911 fclose \%c # Close the file
912 echo Total for "\%1" : \m(total) # Print the result
915 The syntax of the FSEEK command in this example indicates that each
916 search should start relative to the current file line. Since Kermit
917 is an interpretive language, FSEEK is a lot faster than FREAD'ing
918 each line and checking it for the target, especially for big files.
919 An especially handy use for FSEEK is for use with potentially huge
920 sequentially timestamped logs, to seek directly to the date-time
921 where you want to start processing. Some other improvements for the
922 FOPEN/FREAD/FWRITE/FCLOSE family of commands are included also
923 (perfomance, bug fixes, convenience features), listed in the
924 [70]change log. (Prior to 9.0.299 Alpha.02, the FSEEK /FIND:
925 command always started from the top.)
926 * SET SESSION-LOG TEXT now strips out ANSI escape sequences from the
928 * For interacting with POP servers over clear-text or SSL-secured
930 + New SSL and TLS "raw" connections (no Telnet protocol).
931 + New INPUT command options for reading and capturing (perhaps
932 while scanning) continuous incoming text, such as INPUT
933 /NOWRAP (explained [71]HERE).
934 + New \femailaddress() command to extract the e-mail address
935 from an Internet mail message To: or From: line, used in
936 fetching mail from POP servers.
937 + Improved date parsing commands and functions for parsing the
938 different date formats that can appear in e-mail.
939 + Production scripts for fetching mail from a secure POP server,
941 * Various features added to make Kermit more useful for writing CGI
942 scripts such as INPUT /COUNT:n to INPUT exactly n characters
943 (useful for reading form data).
944 * New \fpictureinfo() function for getting orientation and dimensions
945 of JPG and GIF images, described [73]HERE.
946 * New \fgetpidinfo() function for testing whether a given process
948 * \fkwdvalue() function fixed to allow multiword values.
949 * New function \fcount(s1,s2) to tell the number of occurrences of s1
951 * New \flopx() function returns rightmost field from string (such as
953 * New function \ffunction(s1) to tell whether a built-in s1 function
955 * New \fsqueeze(s1) function removes leading and trailing whitespace
956 from string s1, changes tabs to spaces, squeezing each run of
957 repeated whitespace characters to a single space (Alpha.02).
958 * Compact substring notation: \s(somestring[12:18]) is the same as
959 \fsubstring(\m(somestring),12,18), i.e. the substring starting at
960 position 12, 18 charcters long. \s(somestring[12_18]) means
961 characters 12 through 18 of the string (7 characters).
962 * The string indexing functions now accept an optional trailing
963 argument specifying the occurrence number of the target string.
964 Likewise, \fword() can fetch words from the right as well as the
966 * The COPY command in Unix C-Kermit has a new /PRESERVE switch,
967 equivalent to Unix "cp -p".
968 * ASKQ /ECHO:c can be used to make the characters the user types echo
969 as the character c, e.g. asterisk when typing a password.
970 * IF LINK filename to test if the filename is a symlink.
971 * Ctrl-K, when typed at the command parser, replaces itself with most
972 recently entered file specification.
973 * In Unix, the ability to log a terminal session to a serial port,
974 for use with speaking devices or serial printers; described
975 [74]HERE. Also for the same purpose, SET SESSION-LOG
976 NULL-PADDED-LINES for a speech synthesizer than needed this.
977 * Adaptation to OpenSSL 0.9.8 and 1.0.0.
978 * Lifted the restriction on having a remote Kermit program send
979 REMOTE commands to the local. A very big ex-client needed to be
980 able to do this (branches would connect to headquarters and upload
981 files; HQ would then download patches, a REMOTE HOST command was
982 necessary to allow the remote headquarters machines to install the
983 patches on the local client; of course the client first has to
984 ENABLE HOST because this is a risky scenario). The reason for the
985 restriction was that the server, upon receiving any REMOTE command
986 would send the results (output) back to the client as a file
987 transfer with "destination screen", but of course the remote has no
989 * [75]MIME synonyms for character-set names were introduced in
990 Alpha.05. Nobody seemed to notice that after that, character-set
991 selection didn't work at all. Anyway, now it's fixed.
992 * Added XMESSAGE, which is to MESSAGE (Alpha.03) as XECHO is ECHO: it
993 outputs a string with no line terminator DEBUG MESSAGE is ON.
994 * Fixed \recurse() to not dump core when invoked with no arguments.
995 * Improved text for HELP FUNCTION SPLIT and HELP FUNCTION WORD.
996 * Patches for Debian 6.0 "Squeeze" from Ian Beckwith.
997 * \fcontents(\&a[3]) got an error if the array was declared but its
998 dimension was less than 3. Now it simply returns and empty string.
999 * \fsplit(), when parsing lines from CSV and TSV files, was treating
1000 backslash in the data the same way it treats backslash in Kermit
1001 commands. This was fixed to treat backslash like any other
1003 * Builds for Solaris 9 and later now use streams ptys rather then the
1004 old BSD-style ptys. Thanks to Gary Mills for this one, who noticed
1005 that he couldn't have more than 48 C-Kermit SSH sessions going at
1006 once and figured out why.
1007 * As noted [76]below DES encryption is being retired from many
1008 platforms and libraries that once used it. I changed the Solaris
1009 and Linux OpenSSL builds to account for this by testing for it. I
1010 probably should also add a OMITDES option to omit DES even if it is
1011 installed, but "KFLAGS=-UCK_DES" seems to do the job for now.
1012 * I changed the Linux build to test for the OpenSSL version (like the
1013 Solaris version already did), rather than assuming OpenSSL 0.9.7.
1014 * A couple minor changes for Tru64 Unix 5.1B from Steven Schweda but
1015 we still have some trouble on that platform. As a workaround "make
1016 osf1" can be used there.
1017 * Unix makefile and man page are now included in the Zip
1019 * \fjoin(), which is the inverse function of fsplit() now accepts CSV
1020 and TSV as a second argument, to transform an array into a
1021 comma-separated or tab-separated value list, as described [77]HERE.
1022 * Even in 2010, Unix distributions continue to change their UUCP
1023 lockfile conventions. Alpha.08 contains support from Joop Boonen
1024 for OpenSuse >= 11.3 and recent Debian, which no longer have
1025 baudboy.h, which first appeared in Red Hat 7.2 in 2003.
1026 * From Lewis McCarthy:
1028 Based on code inspection, C-Kermit appears to have an SSL-related
1029 security vulnerability analogous to that identified as CVE-2009-3767
1031 [78]http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3767).
1033 I'm attaching a patch for this issue relative to the revision of
1034 ck_ssl.c obtained from a copy of
1035 [79]http://www.columbia.edu/kermit/ftp/test/tar/x.zip downloaded on
1036 2010/07/30, which I believe is the latest.
1037 When this flaw was first widely publicized at last year's Black Hat
1038 conference, it was claimed that some public certificate authorities
1039 had indeed issued certificates that could be used to exploit this
1040 class of vulnerability. As far as I know they have not revealed
1041 specifically which public CA(s) had been found issuing such
1042 certificates. Some references:
1043 + [80]http://www.mseclab.com/?p=180
1044 + [81]http://www.theregister.co.uk/2009/07/30/universal_ssl_cert
1047 * Peter Eichhorn reported that "RENAME ../x ." didn't work; fixed
1049 * If only one file is FOPEN'd, FCLOSE given with no arguments would
1050 close it; this was a "convenience feature" that turned out to be
1051 dangerous. For safety FCLOSE has to require a specific channel
1052 number or the word ALL.
1053 * Added \fstrcmp(s1,s2,case,start,length), which has the advantage
1054 over IF EQU,LGT,LLT that case senstivity can be specified as a
1055 function arg, and also substrings can be specified.
1056 * Fixed a subtle flaw in the [82]CSV feature that was added in
1057 Alpha.06, namely that if the last item in a comma separated list
1058 was enclosed within doublequotes with a trailing space after the
1059 closing doublequote, a spurious empty final element would be
1060 created in the result array.
1061 * New built-in functions:
1063 \fcvtcsets(string,cs1,cs2)
1064 Function to convert a string from one character set to
1067 \fdecodehex(string[,prefix])
1068 Function to decode a string containing hex escapes.
1070 \fstringtype(string)
1071 Function to tell whether a string is 7-bit, 8-bit, or
1074 For the motivation for these features and an application that uses
1075 them to analyze web logs, see the Weblog script below.
1076 * MIME Character-Set Names: A new equivalence between MIME names and
1077 Kermit names for character sets, with a new table showing the
1078 supported sets [83]HERE (this feature is also illustrated in the
1082 Lazy IF Conditions: Third, now you can do this:
1083 define foo some number
1087 define foo some number
1090 Of course the old way still works too. But watch out because if the
1091 variable name is the same as a symbolic IF condition (for example
1092 COUNT), it won't do what you expected. (IF COUNT was used for loop
1093 control in early versions of MS-DOS Kermit, before it got true FOR
1094 and WHILE loops; it was added to C-Kermit for compatibility, and it
1095 can't be removed because it could break existing scripts).
1096 * Escape sequences are now stripped from text-mode session logs not
1097 only in CONNECT sessions but also in whatever is logged by the
1098 INPUT command; described in the [84]next section.
1099 * New commands for selectively issuing progress or debugging messages
1100 from scripts, also described in the next section.
1101 * Fix from [85]John Dunlap to prevent the fixed packet-timeout
1102 interval from going to an unexpected value.
1103 * Alpha.04 fixes a problem with FTP connections made from 64-bit Unix
1104 platforms. All the other changes in this section were to Alpha.03.
1105 * Relaunching a closed SSH connection with the CONNECT command is now
1106 possible, as it always has been with Telnet and other connection
1107 types; suggested by Peter Eichhorn (needs testing).
1108 * A symbol conflict fixed that prevented successful build on
1110 * Fixes from Christian Corti for building on SunOS 4.1.
1111 * New aixg target for building on AIX with gcc.
1112 * New aix+ibmssl target. This is nice because the IBM-supplied SSL
1113 libraries and header files are in a known location; no need to
1114 [87]set environment variables giving their locations.
1115 * "Large File Support" is now included by default on Alpha and IA64
1116 hardware on VMS 7.3 and later, and it should work much better than
1118 * Kermit's internal FTP client is now included by default in any
1119 build that also includes TCP/IP networking. At present, the FTP
1120 client seems to work well for binary-mode transfers; text (ASCII)
1121 mode transfers still need some work. In builds that also include
1122 Secure Sockets Layer (SSL) security (next item) the FTP client
1123 should be able to make securely authenticated and encrypted
1125 * In network builds that request OpenSSL support, e.g.:
1127 $ @ckvker "" "" "CK_SSL"
1128 the OpenSSL version is detected automatically and the appropriate
1129 compile-time options are emitted (such as
1130 OPENSSL_DISABLE_OLD_DES_SUPPORT).
1131 * Preliminary / limited support for the ODS-5 file system on VMS 7.2
1132 and later, Alpha and Itanium only (needs testing): Filenames can be
1133 mixed case and can be longer.
1134 * Support for older and older VMS versions.
1135 * In the VMS build procedure, CKVKER.COM, the "i" option in P1 now
1136 means don't include the internal FTP client, and the "f" option
1137 means do not include "Large File" support. Large File support in
1138 VMS really only applies to the file-transfer display and
1139 statistics, which would go out of whack as soon as the byte count
1140 overflowed 31 bits because this is C-Kermit, built with the C
1141 compiler and the C library (runtime system), which did not support
1142 long integers until VMS 7.3.
1143 * The [88]LISP Operator ROUND now takes an optional second argument
1144 that specifies the number of places to round to, e.g.
1145 (ROUND dollars 2) rounds dollars to 2 decimal places.
1146 * Improved pattern matching in many commands for both strings and
1148 * Various minor new features, plus numerous bug fixes and speedups.
1152 A top priority for new Kermit software releases has always been
1153 backwards compatibility. A script written for a previous Kermit release
1154 should run the same way in the new release.
1156 There's one exception this time. The [89]\fsplit() function is
1157 incredibly handy, it can do almost anything, up to and including
1158 parsing a LISP program (the underlying code is the basis of the
1159 [90]S-Expression interpreter). But did you ever try to use it to parse
1160 (say) a Tab-Separated-List (TSV file) or Comma-Separated-List (CSV)? It
1161 works as expected as long as the data contains only 7-bit characters.
1162 But if your data contains (say) Spanish or German or Russian text
1163 written in an 8-bit character set such as ISO 8859-1, every 8-bit
1164 character (any value 128-255) is treated as a break character. This is
1165 fixed in C-Kermit 9.0 by treating all 8-bit bytes as "include"
1166 characters rather than break characters, a total reversal of past
1167 behavior. I don't think it will affect anyone though, because if this
1168 had happened to anyone, I would have heard about it!
1170 Since most standard 8-bit character sets have control characters in
1171 positions 128-160, it might have made sense to keep 128-160 in the
1172 break set, but with the proliferation of Microsoft Windows code pages,
1173 there is no telling which 8-bit character is likely to be some kind of
1174 text, e.g. "smart quotes" or East European or Turkish accented letters.
1176 What's Not In C-Kermit 9.0
1178 Some large projects that were contemplated have not been done,
1180 * IPv6. Honestly, there has been zero demand for this, and it would
1181 be a lot of work and disruption to the code base. Volunteers
1182 welcome, I guess. It could be a CS project.
1183 * A database interface - MySQL or ODBC. For this one, there is some
1184 demand but I haven't had a chance to even look into it.
1185 * There's a looming issue with DES encryption; major vendors are
1186 removing it from their platforms, starting with Apple in Mac OS X
1187 10.6, with Microsoft to follow suit. A secure version of Kermit can
1188 be built without DES, but in limited testing successful connections
1189 were spotty (e.g. with Kerberos 5).
1190 * Cleaning up the Unix makefile. It has 25 years' worth of targets in
1191 it. It is very likely safe to remove most of them, since (a) most
1192 old platforms have gone away by now, or have been upgraded, due to
1193 hacking vulnerabilities; (b) the market has consolidated
1194 considerably; and (c) most of the new features of C-Kermit 9.0,
1195 such as large files, won't be of any use on older platforms and
1196 previous C-Kermit versions will remain available.
1197 * Packages. Everybody wants an install package custom made for their
1198 own computer, Linux RPMs being the prime example but far from the
1199 only one. These will come, I suppose (especially with some Linux
1200 sites having a policy against installing any application that does
1201 not come as an RPM). In the meantime, here's a page that describes
1202 some Kermit-specific issues in package construction:
1203 [91]ckpackages.html.
1206 Using External File-Transfer Protocols on Secure Connections
1208 After C-Kermit 8.0.212 Dev.27 (2006/12/22), I spent a big chunk of time
1209 trying to solve a particular problem that some of you have complained
1210 about and others might be familiar with: If you use C-Kermit to make a
1211 secure Telnet connection to another host (e.g. with Telnet SSL/TLS,
1212 Kerberos, or SRP) and then attempt to transfer a file using an external
1213 protocol such as Zmodem, it doesn't work.
1215 That's because as coded (through 8.0.211), C-Kermit simply starts the
1216 external protocol in a fork with its standard i/o redirected to the
1217 connection. This completely bypasses the encryption and decryption that
1218 is done by C-Kermit itself, and of course it doesn't work. The same
1219 thing occurs if you use the REDIRECT command. The routine that handles
1220 this is ttruncmd() in ckutio.c.
1222 In order to allow (say) Zmodem transfers on secure connections, it is
1223 necessary for C-Kermit to interpose itself between the external Zmodem
1224 program and the connection, decrypting the incoming stream before
1225 feeding it to Zmodem and encrypting Zmodem's output before sending out
1228 In principal, this is simple enough. We open a pseudoterminal pair
1229 ("master" and "slave") for Zmodem's i/o and we create a fork and start
1230 Zmodem in it; we read from the fork pty's standard output, encrypt, and
1231 send to the net; we read from the net, decrypt, and write to the fork
1232 pty's standard input.
1234 In practice, it's not so simple. First of all, pseudoterminals (ptys)
1235 don't seem to interface correctly with certain crucial APIs, at least
1236 not in the OS's I have tried (Mac OS X, Linux, NetBSD, etc), such as
1237 select(). And i/o with the pty often - perhaps always - fails to
1238 indicate errors when they occur; for example, when the fork has exited.
1240 But, even after coding around the apparent uselessness of select() for
1241 multiplexing pty and net, and using various tricks to detect when the
1242 external protocol exits and what its exit status is, I'm still left
1243 with a show-stopping problem: I just simply can not download (receive)
1244 a file with Zmodem, which is the main thing that people would probably
1245 want to do. I can send files just fine, but not receive. The incoming
1246 stream is delivered to Zmodem (to the pty slave) but upon arrival at
1247 the Zmodem process itself, pieces are always missing and/or corrupt.
1248 Yet I can receive files just fine if I use Kermit itself (C-Kermit or
1249 G-Kermit) as the external protocol, rather than Zmodem.
1251 I can think of two reasons why this might be the case:
1253 1. Zmodem sends all 8-bit bytes and control codes in the clear, and
1254 maybe the pty is choking on them because it thinks it is a real
1257 But Zmodem puts its controlling terminal into raw mode. And C-Kermit
1258 puts the pty into raw mode too, just for good measure. If any 0xFF
1259 codes are in the Zmodem data stream, and it's a Telnet session, Kermit
1260 does any needed byte stuffing/unstuffing automatically. Anyway, if I
1261 tell Zmodem to prefix everything, it makes no difference.
1263 2. Zmodem is a streaming protocol and perhaps the pty driver can't
1264 keep up with a sustained stream of input at network speeds. What
1265 would be the method of flow control?
1267 I can vary the size of the i/o buffers used for writing to the pty, and
1268 get different effects, but I am not able to get a clean download, no
1269 matter what buffer size I use. write()'ing to the pty does not return
1270 an error, and I can't see the errors because they happen on the master
1271 side. It's as if the path between the pty slave and master lacks flow
1272 control; I deliver a valid data stream to the pty slave and the master
1273 gets bits and pieces. This impression is bolstered somewhat by the
1274 "[92]man 7 pty" page in HP-UX, which talks about some special modes for
1275 ptys that turn off all termio processing and guarantee a
1276 flow-controlled reliable stream of bytes in both directions - a feature
1277 that seems to be specific to HP-UX, and exactly the one we need
1280 Well, in Pass One I used C-Kermit's existing pty routines from
1281 ckupty.[ch], which are well-proven in terms of portability and of
1282 actually working. They are currently used by SET HOST /PTY for making
1283 terminal connections to external processes. But these routines are
1284 written on the assumption that the pty is to be accessed interactively,
1285 and maybe they are setting the fork/pty arrangement up in such a way
1286 that that's not suitable for file transfer. The Pass One routine is
1287 called xttptycmd() in ckutio.c.
1289 So in Pass Two I made a second copy of the routine, yttptycmd(), that
1290 manages the pty and fork itself, so all the code is in one place and
1291 it's simple and understandable. But it still doesn't work for Zmodem
1292 downloads. In this routine, I use openpty() to get the pty pair, which
1293 is not portable, so I can have access to both the master and slave pty
1294 file descriptors. This version can be used only a platforms that have
1295 openpty(): Linux, Mac OS X, NetBSD, etc.
1297 In Pass Three, zttptycmd(), I tried using pipes instead of ptys, in
1298 case ptys are simply not up to this task (but that can't be true
1299 because if I make a Telnet or SSH connection into a host, I can send
1300 files to it with Zmodem, and the remote Zmodem receiver is, indeed,
1301 running on a pty). But pipes didn't work either.
1303 In Pass Four, I extracted the relevant routines into a standalone
1304 program based on yttptycmd() (the openpty() version, for simplicity),
1305 which I tested on Mac OS X, the idea being to rule out any
1306 "environmental" effects of running inside the C-Kermit process. There
1307 was no difference -- Kermit transfers (with C-Kermit itself as the
1308 external protocol) worked; Zmodem transfers (neither sz or lsz) did
1311 Well, it's a much longer story. As the external protocol, I've tried
1312 rzsz, crzsz, and lrzsz. We know that some of these have quirks
1313 regarding standard i/o, etc, which is one of the reasons for using ptys
1314 in the first place, and i/o does work - just not reliably. Anyway, the
1315 1100 lines or so of [93]ckc299.txt, starting just below where it says
1316 "--- Dev.27 ---" tell the full story. At this point I have to give up
1317 and move on; it might be more productive to let somebody else who has
1318 more experience with ptys take a look at it - if indeed anyone still
1319 cares about being able to do Zmodem transfers over secure Telnet
1322 C-Kermit 9.0 contains the three new routines (and some auxiliary ones),
1323 but they are not compiled or called unless you build it specially:
1325 make targetname KFLAGS=-DXTTPTYCMD (builds with xttptycmd())
1326 make targetname KFLAGS=-DYTTPTYCMD (builds with yttptycmd())
1327 make targetname KFLAGS=-DZTTPTYCMD (builds with zttptycmd())
1329 These are all in [94]ckutio.c. As noted, the second one works only for
1330 Linux, FreeBSD, NetBSD, and Mac OS X, because it uses non-POSIX,
1331 non-portable openpty(). If you want to try it on some other platform
1332 that has openpty(), you can build it like this:
1334 make targetname "KFLAGS=-DYTTPTYCMD -DHAVE_OPENPTY"
1336 (and let me know, so I can have HAVE_OPENPTY predefined for that
1337 platform too). The best strategy to get this working, I think, would be
1338 to concentrate on yttptycmd(), which is the simpler of the two
1339 pty-based routines. If it can be made to work, then we'll see if we can
1340 retrofit it to use the ckupty.c routines so it will be portable to
1343 By the way, if you build with any of [XYZ]TTPTYCMD defined, then the
1344 selected routine will always be used in place of ttruncmd(). This is to
1345 allow testing on all kinds of connections, not just secure ones, in
1346 both local and remote mode. Once the thing works, if it ever does, I'll
1347 add the appropriate tests and/or commands.
1349 By default, in the initial test release, C-Kermit 9.0 uses ttruncmd()
1350 on serial connections and ttyptycmd() on network connections. Even when
1351 a network connection is not encrypted, Kermit still needs to handle the
1352 network protocol, e.g. the quoting of 0xff bytes on Telnet connections.
1354 Demonstration: Fetch Mail from POP Server Secured by SSL
1356 [95]pop.ksc is a fully elaborated production script for fetching one's
1357 mail from a POP3 server over a connection secured by SSL. For
1358 explanation and documentation, [96]CLICK HERE. [97]mailcheck is a
1359 wrapper for the pop.ksc script, which collects your password one time,
1360 and then checks for new mail every 5 minutes (or other selected
1361 interval) and calls pop.ksc to fetch it if there is any.
1363 Demonstration: HP Switch Configuration Backup
1365 A common use for Kermit software is to make automated backups of the
1366 configuration of network switches and routers, such as those made by
1367 Cisco or Hewlett-Packard (although [98]tftp can be used for this, it is
1368 not available in all such devices; Kermit, however, works with those
1369 that have tftp as well as those that don't).
1371 Typically a backup can be done by making a Telnet, SSH, or serial
1372 connection to the device with Kermit and giving a command such as "show
1373 config" at the command-line prompt of the device with Kermit's session
1374 log activated. The result is a list of the commands that were used to
1375 establish the current configuration, suitable for feeding back to the
1376 device's console (e.g. with C-Kermit's TRANSMIT command) to reestablish
1377 the same configuration or to duplicate it on another device.
1379 At an HP installation it was noted, however, that while the HP switches
1380 (various ProCurve models) produced the desired list of commands, they
1381 were interspersed with escape sequences for special effects, thus
1382 rendering the recorded sessions unsuitable for feeding back into the
1385 C-Kermit 9.0 introduces a new feature to strip the offending sequences
1386 out of a session log, leaving just the text. The command SET
1387 SESSION-LOG TEXT activates this feature. In C-Kermit 9.0 Alpha.02 and
1388 earlier, escape sequence stripping occurred only while logging
1389 interactive (CONNECT) sessions; beginning with Alpha.03 it is done also
1390 for data that is read by INPUT commands and therefore works for scripts
1393 A sample HP Switch Configuration Backup script is [99]HERE, and its
1394 data file is [100]HERE. This script also illustrates some other new
1395 features of Alpha.03:
1398 This lets you put debugging messages in your script that can be
1399 displayed or not, according to SET DEBUG MESSAGE (below). This
1400 way you don't have to change your script for debugging. Hint:
1401 In Unix, invoke the script like this:
1403 $ DEBUG=1 scriptname arg1 arg2...
1405 and then include the following command in your script:
1407 if defined \$(DEBUG) set debug message on
1410 Like MESSAGE but prints the text with no line terminator, so it
1411 can be continued by subsequent messages.
1413 SET DEBUG MESSAGE { ON, OFF, STDERR }
1414 ON means MESSAGE commands should print to standard output; OFF
1415 means they shouldn't print anything; STDERR means the messages
1416 should be printed to [101]stderr. DEBUG MESSAGE is OFF by
1417 default, i.e. unless you SET it to ON or STDERR.
1420 Executes the command if SET DEBUG MESSAGE is not OFF.
1422 The \v(lastcommand) variable
1423 This variable contains the previous command. You can use it in
1424 debugging and error message to show (for example) exactly what
1425 the command was that just failed, without having to make a copy
1428 set host somehost.somecompany.com
1429 if fail exit 1 "FATAL - \v(lastcommand)"
1431 which, if the SET HOST command fails, prints "FATAL - set host
1432 somehost.somecompany.com" and then exits with status 1 (which
1433 normally indicates failure).
1435 Demonstration: HP iLO Blade Configuration
1437 [102]THIS DOCUMENT describes a script in production use at Columbia
1438 University for configuring and deploying racks full of HP blade servers
1439 through their "integrated Lights Out" (iLO) management interface,
1440 bypassing the tedious and error-prone process of configuring the
1441 servers one by one through the vendor-provided point-and-click Web
1442 interface, which is ill-suited to configuring large numbers of blades.
1443 The script illustrates some of C-Kermit 9.0's new features; source code
1444 is available through the link. The code is apt to change from time to
1445 time as new requirements surface.
1447 Demonstration: IBM/Rolm/Siemens CBX Management
1449 [103]THIS DOCUMENT describes a suite of scripts (some in production,
1450 some in development) used to manage the Columbia campus 20,000-line
1451 main telephone switch, along with about 10 satellite switches at
1452 off-campus locations. These switches are 1980s technology*, their
1453 management consoles are serial ports. Access is via Telnet to reverse
1454 terminal servers. The scripts allow for interactive sessions as well as
1455 automatic production (and in some cases formatting) of different
1456 reports required by different groups at different intervals. These
1457 scripts replace a whole assortment of ad-hoc ProComm ASPECT scripts
1458 that were scattered all over the place, with passwords embedded. The
1459 new scripts are intended to be run from a centralized server where
1460 there is a single well-secured configuration file, and where they can
1461 be used on demand, or in cron jobs. They are modular so code
1462 duplication is minimal.
1463 __________________________
1464 * Of course the University is deploying new technology but the but the
1465 old system will be used in parallel for some time to come.
1467 Demonstration: CSV and TSV Files
1471 * [104]Reading a CSV or TSV Record and Converting it to an Array
1472 * [105]Using \fjoin() to create a Comma- or Tab-Separated Value List
1474 * [106]Using CSV or TSV Files
1476 Comma-Separated Value (CSV) format is commonly output by spreadsheets
1477 and databases when exporting data into plain-text files for import into
1478 other applications. Here are the details:
1480 Comma-Separated List Syntax
1482 1. Each record is a series of fields.
1483 2. Records are in whatever format is used by the underlying file
1484 system for lines of text.
1485 3. Fields within records are separated by commas, with zero or more
1486 whitespace characters (space or tab) before and/or after the comma;
1487 such whitespace is considered part of the separator.
1488 4. Fields with imbedded commas must be enclosed in ASCII doublequote
1490 5. Fields with leading or trailing spaces must be enclosed in ASCII
1492 6. Any field may be enclosed in ASCII doublequotes.
1493 7. Fields with embedded doublequotes must be enclosed in doublequotes
1494 and each interior doublequote is doubled.
1498 aaa, bbb, has spaces,,"ddd,eee,fff", " has spaces ","Muhammad ""The Greatest"" A
1501 The first two are regular fields. The second is a field that has an
1502 embedded space but in which any leading or trailing spaces are to be
1503 ignored. The fourth is an empty field, but still a field. The fifth is
1504 a field that contains embedded commas. The sixth has leading and
1505 trailing spaces. The last field has embedded quotation marks.
1507 Prior to C-Kermit 9.0 Alpha.06, C-Kermit did not handle CSV files
1508 according to the specification above. Most seriously, there was no
1509 provision for a separator to be surrounded by whitespace that was to be
1510 considered part of the separator. Also there was no provision for
1511 quoting doublequotes inside of a quoted string.
1513 Reading a CSV record
1515 Now the \fsplit() function can handle any CSV-format string if you
1516 include the symbolic include set "CSV" as the 4th parameter. To
1517 illustrate, this program:
1520 echo [\fcontents(\%1)]
1521 .\%9 := \fsplit(\fcontents(\%1), &a, \44, CSV)
1522 for \%i 1 \%9 1 { echo "\flpad(\%i,3). [\&a[\%i]]" }
1527 xx { aaa,,ccc," with spaces ",zzz }
1528 xx { "1","2","3","","5" }
1529 xx { this is a single field }
1530 xx { this is one field, " and this is another " }
1531 xx { name,"Mohammad ""The Greatest"" Ali", age, 67 }
1532 xx { """field enclosed in doublequotes""" }
1535 gives the following results:
1547 [ aaa,,ccc," with spaces ",zzz ]
1554 [ "1","2","3","","5" ]
1561 [ this is a single field ]
1562 1. [this is a single field]
1564 [ this is one field, " and this is another " ]
1565 1. [this is one field]
1566 2. [ and this is another ]
1568 [ name,"Mohammad ""The Greatest"" Ali", age, 67 ]
1570 2. [Mohammad "The Greatest" Ali]
1574 [ """field enclosed in doublequotes""" ]
1575 1. ["field enclosed in doublequotes"]
1578 The separator \44 (comma) must still be specified as the break set (3rd
1579 \fsplit() parameter). When "CSV" is specified as the include set:
1580 * The Grouping Mask is automatically set to 1 (which specifies that
1581 the ASCII doublequote character (") is used for grouping;
1582 * The Separator Flag is automatically set to 1 so that adjacent field
1583 separators will not be collapsed;
1584 * All bytes (values 0 through 255) other than the break character are
1585 added to the include set;
1586 * Any leading whitespace is stripped from the first element unless it
1587 is enclosed in doublequotes;
1588 * Any trailing whitespace is trimmed from the end of the last element
1589 unless it is enclosed in doublequotes;
1590 * If the separator character has any spaces or tabs preceding it or
1591 following it, they are ignored and discarded;
1592 * The separator character is treated as an ordinary data character if
1593 it appears in a quoted field;
1594 * A sequence of two doublequote characters ("") within a quoted field
1595 is converted to a single doublequote.
1597 There is also a new TSV symbolic include set, which is like CSV except
1598 without the quoting rules or the stripping of whitespace around the
1599 separator because, by definition, TSV fields do not contain tabs.
1601 Of course you can specify any separator(s) you want with either the
1602 CSV, TSV, or ALL symbolic include sets. For example, if you have a TSV
1603 file in which you want the spaces around each Tab to be discarded, you
1606 \fsplit(variable, &a, \9, CSV)
1610 The new symbolic include sets can also be used with \fword(), which is
1611 just like \fsplit() except that it retrieves the nth word from the
1612 argument string, rather than an array of all the words. In C-Kermit you
1613 can get information about these or any other functions with the HELP
1614 FUNCTION command, e.g.:
1616 C-Kermit> help func word
1618 Function \fword(s1,n1,s2,s3,n2,n3) - Extracts a word from a string.
1620 n1 = word number (1-based) counting from left; if negative, from right.
1621 s2 = optional break set.
1622 s3 = optional include set (or ALL, CSV, or TSV).
1623 n2 = optional grouping mask.
1624 n3 = optional separator flag:
1625 0 = collapse adjacent separators;
1626 1 = don't collapse adjacent separators.
1628 \fword() returns the n1th "word" of the string s1, according to the
1629 criteria specified by the other parameters.
1631 The BREAK SET is the set of all characters that separate words. The
1632 default break set is all characters except ASCII letters and digits.
1633 ASCII (C0) control characters are treated as break characters by default,
1634 as are spacing and punctuation characters, brackets, and so on, and
1635 all 8-bit characters.
1637 The INCLUDE SET is the set of characters that are to be treated as
1638 parts of words even though they normally would be separators. The
1639 default include set is empty. Three special symbolic include sets are
1642 ALL (meaning include all bytes that are not in the break set)
1643 CSV (special treatment for Comma-Separated-Value records)
1644 TSV (special treatment for Tab-Separated-Value records)
1646 For operating on 8-bit character sets, the include set should be ALL.
1648 If the GROUPING MASK is given and is nonzero, words can be grouped by
1649 quotes or brackets selected by the sum of the following:
1651 1 = doublequotes: "a b c"
1653 4 = apostrophes: 'a b c'
1654 8 = parentheses: (a b c)
1655 16 = square brackets: [a b c]
1656 32 = angle brackets: <a b c>
1658 Nesting is possible with {}()[]<> but not with quotes or apostrophes.
1661 Word number n1, if there is one, otherwise an empty string.
1668 Using \fjoin() to create Comma- or Tab-Separated Value Lists from Arrays
1670 In C-Kermit 9.0, \fsplit()'s inverse function, [107]\fjoin() received
1671 the capability of converting an array into a comma-separated or a
1672 tab-separated value list. Thus, given a CSV, if you split it into an
1673 array with \fsplit() and then join the array with \fjoin(), giving each
1674 function the new CSV parameter in the appropriate argument position,
1675 the result will be will be equivalent to the original, according to the
1676 CSV definition. It might not be identical, because if the result had
1677 extraneous spaces before or after the separating commas, these are
1678 discarded, but that does not affect the elements themselves. The new
1679 syntax for \fjoin() is:
1682 Given the array \&a[] or any other valid array designator, joins
1683 its elements into a comma-separated list according to the
1684 [108]rules listed above.
1687 Joins the elements of the given array into a tab-separated list,
1688 also described above.
1690 [109]Previous calling conventions for \fjoin() are undisturbed,
1691 including the ability to specify a portion of an array, rather than the
1694 declare \&a[] = 1 2 3 4 5 6 7 8 9
1695 echo \fjoin(&a[3:7],CSV)
1698 Using \fsplit() and \fjoin() it is now possible to convert a
1699 comma-separated value list into a tab-separated value list, and vice
1700 versa (which is not a simple matter of changing commas to tabs or vice
1703 Applications for CSV Files
1705 Databases such as MS Access or MySQL can export tables or reports in
1706 CSV format, and then Kermit can read the resulting CSV file and do
1707 whatever you like with it; typically something that could not be done
1708 with the database query language itself (or that you didn't know how to
1709 do that way): create reports or datasets based on complex criteria or
1710 procedures, edit or modify some fields, etc, and then use \fjoin() to
1711 put each record back in CSV form so it can be reimported into a
1712 spreadsheet or database.
1714 Here is a simple example in which we purge all records of customers who
1715 have two or more unpaid bills. The file is sorted so that each license
1716 purchase record is followed by its annual maintenance payment records
1717 in chronological order.
1719 #!/usr/local/bin/kermit
1720 .filename = somefile.csv # Input file in CSV format
1721 fopen /read \%c \m(filename) # Open it
1722 if fail exit # Don't go on if open failed
1723 copy \m(filename) ./new # Make a copy of the file
1725 .oldserial = 00000000000 # Multiple records for each serial number
1726 .zeros = 0 # Unpaid bill counter
1729 fread /line \%c line # Get a record
1730 if fail exit # End of file
1731 .n := \fsplit(\m(line),&a,\44,CSV) # Split the fields into an array
1732 if not equ "\m(oldserial)" "\&a[6]" { # Have new serial number?
1733 # Remove all records for previous serial number
1734 # if two or more bills were not paid...
1736 grep /nomatch \m(oldserial) /output:./new2 ./new
1739 .oldserial := \&a[6] # To detect next time serial number changes
1740 .zeros = 0 # Reset unpaid bill counter
1742 if equ "\&a[5]" "$0.00" { # Element 5 is amount paid
1743 increment zeros # If it's zero, count it.
1748 Rewriting the file multiple times is inelegant, but this is a quick and
1749 dirty use-once-and-discard script, so elegance doesn't count. The
1750 example is interesting in that it purges certain records based on the
1751 contents of other records. Maybe there is a way to do this directly
1752 with SQL, but why use SQL when you can use Kermit?
1754 Here is the same task but this time no shelling out, and this time we
1755 do change and add some fields and then join the result back into a CSV
1756 record and write it out to a new file. The object is to create a record
1757 for each license that shows not only the date and purchase price of the
1758 license but also the date and amount of the last maintenance payment,
1759 and to add new fields for sorting by anniversary (month and day):
1761 #!usr/local/bin/kermit +
1762 cd ~/somedirectory # CD to appropriate directory
1763 if fail exit 1 # Make sure we did
1764 .filename := \%1 # Filename from command line
1765 if not def filename { # If none give usage message
1766 exit 1 "Usage: \%0: infile [ outfile ]"
1768 fopen /read \%c \m(filename) # Open the input CSV file
1769 if fail exit # Make sure we did
1771 .output := \%2 # Output filename from command line
1772 if not def output { # Supply one if not given
1773 .output := New_\m(filename)
1775 fopen /write \%o \m(output) # Open output file
1776 if fail exit # Check that we did
1778 .serial = 00000000000 # Initialize serial number
1779 .licenses = 0 # and license counter
1781 fread /line \%c line # First line is column labels
1782 if fail exit # Check
1783 fwrite /line \%o "\m(line),AMM_DD,AYYYY" # Write new labels line
1785 # Remaining lines are license purchases (K95B) followed by zero or more
1786 # maintenance invoices (K95BM) for each license.
1788 .datepaid = 00/00/0000 # Initialize last maint payment date
1789 .amtpaid = $0.00 # Initialize last maint payment amount
1790 set flag off # For remembering we're at end of file
1791 while not flag { # Loop to read all records
1792 fread /line \%c line # Read a record
1793 if fail set flag on # If EOF set flag for later
1794 .n := \fsplit(\m(line),&a,\44,CSV) # Break record into array
1795 if ( flag || equ "\&a[3]" "K95B" ) { # License or EOF
1796 if fail exit 1 "FAILED: \v(lastcommand)"
1797 if licenses { # If this is not the first license
1798 .\&x[5] := \m(amtpaid) # Substitute most recent amount paid
1799 .\&x[21] := \m(datepaid) # Substitute most recent date paid
1800 void \fsplit(\&x[18],&d,/) # Break up original (anniversary) date
1801 # and put mm_dd and yyyy in separate fields for sorting...
1802 fwrite /line \%o "\fjoin(&x,CSV),\flpad(\&d[1],2,0)_\flpad(\&d[2],2,
1804 if fail exit 1 WRITE # Check for error
1805 xecho . # Show progress as one dot per record
1807 if flag break # We're at EOF so we're finished
1808 increment licenses # New license - count it
1809 array copy &a &x # Keep this record while reading next
1810 .serial := \&a[6] # Remember serial number
1811 .datepaid = 00/00/0000 # Initial maintenance payment date
1812 .amtpaid = $0.00 # and amount
1813 continue # and go back to read next record
1815 if not eq "\m(serial)" "\&a[6]" { # Catch out-of-sequence record
1817 echo "SEQUENCE: \m(serial)..\&a[6]: \&a[7] [\&a[1]]"
1820 if equ "\&a[5]" "" .\&a[5] = $0.00 # If amount is empty make it $0.00
1821 if not equ "\&a[5]" "$0.00" { # If amount is not $0.00
1822 .datepaid := \&a[21] # remember date paid
1823 .amtpaid := \&a[5] # and amount paid
1826 fclose ALL # Done - close all files and exit
1830 The result imports back into Excel, where it can be sorted, formatted,
1831 or otherwise manipulated as desired.
1833 Using CSV Files: Extending Kermit's Data Structures
1835 Now that we can parse a CSV record, what would we do with a CSV file -
1836 that is, a sequence of records? If we needed all the data available at
1837 once, we would want to load it into a matrix of (row,column) values.
1838 But Kermit doesn't have matrices. Or does it?
1840 Kermit has several built-in data types, but you can invent your own
1841 data types as needed using Kermit's macro feature:
1843 define variablename value
1847 define alphabet abcdefghijklmnopqrstuvwxyz
1849 This defines a macro named alphabet and gives it the value
1850 abcdefghijklmnopqrstuvwxyz. A more convenient notation (added in
1851 C-Kermit 7.0, see [110]Table 2) for this is:
1853 .alphabet = abcdefghijklmnopqrstuvwxyz
1855 The two are exactly equivalent: they make a literal copy the "right
1856 hand side" as the value of the macro. Then you can refer to the macro
1857 anywhere in a Kermit command as "\m(macroname)":
1859 echo "Alphabet = \m(alphabet)"
1861 There is a second way to define a macro, which is like the first except
1862 that the right-hand side is evaluated first; that is, any variable
1863 references or function calls in the right-hand side are replaced by
1864 their values before the result is assigned to the macro. The command
1865 for this is ASSIGN rather than DEFINE:
1867 define alphabet abcdefghijklmnopqrstuvwxyz
1868 assign backwards \freverse(\m(alphabet))
1869 echo "Alphabet backwards = \m(backwards)"
1873 Alphabet backwards = zyxwvutsrqponmlkjihgfedcba
1875 This kind of assignment can also be done like this:
1877 .alphabet = abcdefghijklmnopqrstuvwxyz
1878 .backwards := \freverse(\m(alphabet))
1880 [111]Any command starting with a period is an assignment, and the
1881 operator (= or :=) tells what to do with the right-hand side before
1882 making the assignment.
1884 In both the DEFINE and ASSIGN commands, the variable name itself is
1885 taken literally. It is also possible, however, to have Kermit compute
1886 the variable name. This is done (as described in [112]Using C-Kermit,
1887 2nd Ed., p.457), using parallel commands that start with underscore:
1888 _DEFINE and _ASSIGN (alias _DEF and _ASG). These are just like DEFINE
1889 and ASSIGN except they evaluate the variable name before making the
1890 assigment. For example:
1893 _define \%a\%a\%a 111
1895 would create a macro named ONEONEONE with a value of 111, and:
1899 _assign \%a\%a\%a \m(number)
1901 would create the same macro with the same value, but:
1905 _define \%a\%a\%a \m(number)
1907 would give the macro a value of "\m(number)".
1909 You can use the _ASSIGN command to create any kind of data structure
1910 you want; you can find some examples in the [113]Object-Oriented
1911 Programming section of the [114]Kermit Script Library. In the following
1912 program we use this capability to create a two-dimensional array, or
1913 matrix, to hold the all the elements of the CSV file, and then to
1916 fopen /read \%c data.csv # Open CSV file
1920 .\%m = 0 # Maximum columns
1922 fread /line \%c line # Read a record
1923 if fail break # End of file
1924 .\%n := \fsplit(\m(line),&a,\44,CSV) # Split record into items
1925 incr \%r # Count this row
1926 for \%i 1 \%n 1 { # Assign items to this row of matrix
1927 _asg a[\%r][\%i] \&a[\%i]
1929 if > \%i \%m { .\%m := \%i } # Remember width of widest row
1931 fclose \%c # Close CSV file
1932 decrement \%m # (because of how FOR loop works)
1933 echo MATRIX A ROWS: \%r COLUMNS: \%m # Show the matrix
1935 for \%i 1 \%r 1 { # Loop through rows
1936 for \%j 1 \%m 1 { # Loop through columns of each row
1937 xecho "\flpad(\m(a[\%i][\%j]),6)"
1943 The matrix is called a and its elements are a[1][1], a[1][2], a[1][3],
1944 ... a[2][1], etc, and you can treat this data structure exactly like a
1945 two-dimensional array, in which you can refer to any element by its "X
1946 and Y coordinates". For example, if the CSV file contained numeric data
1947 you could compute row and column sums using simple FOR loops and
1948 Kermit's built-in one-dimensional array data type:
1950 declare \&r[\%r] # Make an array for the row sums
1951 declare \&c[\%m] # Make an array for the column sums
1952 for \%i 1 \%r 1 { # Loop through rows
1953 for \%j 1 \%m 1 { # Loop through columns of each row
1954 increment \&r[\%i] \m(a[\%i][\%j]) # Accumulate row sum
1955 increment \&c[\%j] \m(a[\%i][\%j]) # Accumulate column sum
1959 Note that the sum arrays don't have to be initialized to zero because
1960 Kermit's INCREMENT command treats empty definitions as zero.
1962 Demonstration Scripts for Webmasters
1965 A C-Kermit 9.0 script to build sitemap.xml for a website,
1966 complete with Google image extensions (this is the file used by
1967 webmasters to get their sites crawled and indexed optimally).
1969 [116]The Weblog Script
1970 Reads a web log, extracts the Google searches, normalizes the
1971 search strings, and prints the top 20 searches, along with their
1972 counts. Documented [117]HERE.
1974 [118]The Amazon Script
1975 Reads an Amazon Associate orders report and lists the products
1976 according to the number of orders for each, or the number of
1980 Makes a website from a collecion of JPG images. For explanation
1981 and documentation, [120]CLICK HERE. Requires [121]C-Kermit 9.0
1984 [122]Home [123]Kermit 95 [124]C-Kermit [125]Scripts [126]Current
1985 [127]New [128]FAQ [129]Support
1988 C-Kermit 9.0 / [130]The Kermit Project / [131]Columbia University /
1989 [132]kermit@columbia.edu / [133]validate
1993 1. http://www.columbia.edu/
1994 2. mailto:kermit@columbia.edu
1995 3. http://www.columbia.edu/kermit/index.html
1996 4. http://www.columbia.edu/kermit/k95.html
1997 5. http://www.columbia.edu/kermit/ckermit.html
1998 6. http://www.columbia.edu/kermit/ckscripts.html
1999 7. http://www.columbia.edu/kermit/current.html
2000 8. http://www.columbia.edu/kermit/whatsnew.html
2001 9. http://www.columbia.edu/kermit/faq.html
2002 10. http://www.columbia.edu/kermit/support.html
2003 11. http://www.columbia.edu/kermit/ck90tables.html
2004 12. http://www.amazon.com/gp/product/1555581641?ie=UTF8&tag=aleidmoreldom-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1555581641
2005 13. http://www.columbia.edu/kermit/ckermit.html#download
2006 14. http://www.columbia.edu/kermit/ckermit90.html#LargeFiles
2007 15. http://www.columbia.edu/kermit/ckermit90.html#TestLargeFiles
2008 16. http://www.columbia.edu/kermit/ckermit90.html#Bignums
2009 17. http://www.columbia.edu/kermit/ckermit90.html#force3
2010 18. http://www.columbia.edu/kermit/ckermit90.html#Vareval
2011 19. http://www.columbia.edu/kermit/ckermit90.html#rename
2012 20. http://www.columbia.edu/kermit/ckermit90.html#Other
2013 21. http://www.columbia.edu/kermit/ckermit90.html#Incompatibilities
2014 22. http://www.columbia.edu/kermit/ckermit90.html#NotIn9.0
2015 23. http://www.columbia.edu/kermit/ckermit90.html#LooseEnd
2016 24. http://www.columbia.edu/kermit/ckermit90.html#pop
2017 25. http://www.columbia.edu/kermit/ckermit90.html#HPswitch
2018 26. http://www.columbia.edu/kermit/ckermit90.html#iLO
2019 27. http://www.columbia.edu/kermit/ckermit90.html#Rolm
2020 28. http://www.columbia.edu/kermit/ckermit90.html#CSV
2021 29. http://www.columbia.edu/kermit/ckermit90.html#Otherdemos
2022 30. http://www.columbia.edu/kermit/ck60manual.html
2023 31. http://www.amazon.com/gp/product/B002ACPF9M?ie=UTF8&tag=aleidmoreldom-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=B002ACPF9M
2024 32. http://www.columbia.edu/kermit/ckermit70.html
2025 33. http://www.columbia.edu/kermit/ckermit80.html
2026 34. http://www.columbia.edu/kermit/ckscripts.html
2027 35. http://www.columbia.edu/kermit/dec20.html
2028 36. mailto:fdc@columbia.edu
2029 37. http://www.columbia.edu/kermit/cu-bsd-license.html
2030 38. http://www.columbia.edu/kermit/ckermit90.html#LargeFiles
2031 39. http://www.columbia.edu/kermit/ck90tables.html
2032 40. http://www.columbia.edu/kermit/ck90tables.html
2033 41. http://www.columbia.edu/kermit/ckermit90.html#force3
2034 42. http://www.columbia.edu/kermit/ckermit90.html#Vareval
2035 43. http://www.columbia.edu/kermit/ckrename.html
2036 44. http://www.columbia.edu/kermit/csv.html
2037 45. http://www.columbia.edu/kermit/csetnames.html
2038 46. http://www.columbia.edu/kermit/ckermit90.html#HPswitch
2039 47. http://www.columbia.edu/kermit/ckdaily.html
2040 48. http://www.columbia.edu/kermit/cu-bsd-license.html
2041 49. http://www.opensource.org/
2042 50. http://kermit.columbia.edu/ck90tables.html#LF
2043 51. ftp://kermit.columbia.edu/kermit/utils/bigfile.c
2044 52. http://www.columbia.edu/kermit/ckermit80.html#x9
2045 53. http://www.columbia.edu/kermit/ck90tables.html#LF
2046 54. ftp://kermit.columbia.edu/kermit/scripts/ckermit/easter2
2047 55. http://www.columbia.edu/kermit/em-apex.html
2048 56. http://www.iridium.com/
2049 57. http://science1.nasa.gov/science-news/science-at-nasa/2006/09jan_electrichurricanes/
2050 58. http://www.columbia.edu/kermit/ek.html
2051 59. ftp://kermit.columbia.edu/kermit/ek/simirid/
2052 60. http://www.columbia.edu/kermit/ek.html
2053 61. http://www.columbia.edu/kermit/ckermit70.html#x7.10.10
2054 62. http://www.columbia.edu/kermit/csv.html
2055 63. http://www.columbia.edu/kermit/ckermit70.html#x1.11
2056 64. http://www.columbia.edu/kermit/ckermit70.html
2057 65. http://www.columbia.edu/kermit/ckermit80.html#x9
2058 66. http://www.columbia.edu/kermit/csetnames.html
2059 67. http://www.columbia.edu/kermit/ckdaily.html
2060 68. http://www.columbia.edu/kermit/ckrename.html
2061 69. http://www.columbia.edu/kermit/ftpclient.html
2062 70. http://www.columbia.edu/kermit/ckdaily.html
2063 71. http://www.columbia.edu/kermit/input_nowrap.html
2064 72. http://www.columbia.edu/~fdc/mm/index.html
2065 73. http://www.columbia.edu/kermit/photoalbum.html
2066 74. http://www.columbia.edu/~fdc/kermit/logserial.html
2067 75. http://www.columbia.edu/kermit/csetnames.html
2068 76. http://www.columbia.edu/kermit/ckermit90.html#NotIn9.0
2069 77. http://www.columbia.edu/kermit/csv.html#join
2070 78. http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3767
2071 79. http://www.columbia.edu/kermit/ftp/test/tar/x.zip
2072 80. http://www.mseclab.com/?p=180
2073 81. http://www.theregister.co.uk/2009/07/30/universal_ssl_certificate/
2074 82. http://www.columbia.edu/kermit/csv.html
2075 83. http://www.columbia.edu/kermit/csetnames.html
2076 84. http://www.columbia.edu/kermit/ckermit90.html#HPswitch
2077 85. http://www.columbia.edu/kermit/em-apex.html
2078 86. http://www.freebsd.org/releases/8.0R/announce.html
2079 87. http://www.columbia.edu/kermit/security81.html#x4.2.3
2080 88. http://www.columbia.edu/kermit/ckermit80.html#x9
2081 89. http://www.columbia.edu/kermit/ckermit80.html#x8.7.2
2082 90. http://www.columbia.edu/kermit/ckermit80.html#x9
2083 91. http://www.columbia.edu/kermit/ckpackages.html
2084 92. http://docs.hp.com/en/B9106-90013/pty.7.html
2085 93. http://www.columbia.edu/kermit/test/text/ckc299.txt
2086 94. http://www.columbia.edu/kermit/test/text/ckutio.c
2087 95. http://www.columbia.edu/~fdc/mm/pop
2088 96. http://www.columbia.edu/~fdc/mm/
2089 97. http://www.columbia.edu/~fdc/mm/mailcheck
2090 98. http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
2091 99. http://www.columbia.edu/kermit/ftp/scripts/ckermit/gethpconfig
2092 100. http://www.columbia.edu/kermit/ftp/scripts/ckermit/TestSwitches.txt
2093 101. http://en.wikipedia.org/wiki/Standard_streams
2094 102. http://kermit.columbia.edu/cudocs/ilosetup.html
2095 103. http://www.columbia.edu/kermit/cudocs/cbx.html
2096 104. http://www.columbia.edu/kermit/ckermit90.html#record
2097 105. http://www.columbia.edu/kermit/ckermit90.html#join
2098 106. http://www.columbia.edu/kermit/ckermit90.html#file
2099 107. http://www.columbia.edu/kermit/ckermit80.html#fjoin
2100 108. http://www.columbia.edu/kermit/ckermit90.html#rules
2101 109. http://www.columbia.edu/kermit/ckermit80.html#fjoin
2102 110. http://www.columbia.edu/kermit/ckermit90.html#varasg
2103 111. http://www.columbia.edu/kermit/ckermit70.html#x7.9
2104 112. http://www.amazon.com/gp/product/1555581641?ie=UTF8&tag=aleidmoreldom-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1555581641
2105 113. http://www.columbia.edu/kermit/ckscripts.html#oops
2106 114. http://www.columbia.edu/kermit/ckscripts.html
2107 115. http://www.columbia.edu/kermit/ksitemap.html
2108 116. http://kermit.columbia.edu/ftp/scripts/ckermit/weblog
2109 117. http://www.columbia.edu/kermit/weblog.html
2110 118. http://kermit.columbia.edu/ftp/scripts/ckermit/amazon
2111 119. http://kermit.columbia.edu/ftp/scripts/ckermit/photoalbum
2112 120. http://www.columbia.edu/kermit/photoalbum.html
2113 121. http://www.columbia.edu/kermit/ck90.html
2114 122. http://www.columbia.edu/kermit/index.html
2115 123. http://www.columbia.edu/kermit/k95.html
2116 124. http://www.columbia.edu/kermit/ckermit.html
2117 125. http://www.columbia.edu/kermit/ckscripts.html
2118 126. http://www.columbia.edu/kermit/current.html
2119 127. http://www.columbia.edu/kermit/whatsnew.html
2120 128. http://www.columbia.edu/kermit/faq.html
2121 129. http://www.columbia.edu/kermit/support.html
2122 130. http://www.columbia.edu/kermit/index.html
2123 131. http://www.columbia.edu/
2124 132. mailto:kermit@columbia.edu
2125 133. http://validator.w3.org/check?uri=http%3A%2F%2Fkermit.columbia.edu%2Fck90.html