CODEWITHSUNDEEP
linuxbatch-file
Castro
Castro

Reputation: 1456

Modern Batch Processing in Linux

What tools, languages, and infrastructure do you use for do batch processing in Linux?

I am looking for something that facilitate the tasks of:

Thanks for any help!

Upvotes: 2

Views: 3128

Answers (7)

James Andino
James Andino

Reputation: 25789

Bash is your friend. I would also suggest using Python but thats a bit more preference in my opinion. Also if this is something that needs to be accessed or maintained by slightly less ummmm competent individuals you can create an Interface easily using PHP.

Upvotes: 0

Sebastián Grignoli
Sebastián Grignoli

Reputation: 33472

Have you tried Pentaho Data Integration?

With PDI you can just draw flowcharts instead of programming... It's awesome!

Just kidding.

Upvotes: 0

Dave
Dave

Reputation: 5173

If we're talking about an enterprise implementation, I recommend Load Sharing Facility. This is a system that distributes jobs across a cluster of machines. It allows for queueing of jobs; pausing and stopping jobs; and gives plenty of statistics. You can have one or a few machines be "logon" machines where people submit jobs from and several machines that run the jobs full time.

Upvotes: 0

S.Lott
S.Lott

Reputation: 392050

We use Python.

Process files - the os module does some of this.

Log - the logging module.

Validation - no idea what this means.

Job Controlling (start,strop,reestart a process) - this is complex, ambitious and difficult to define. Some folks use commercial products for this because of the potential auditability issues. Control/M, Tivoli, etc., are good solutions. However, you can build some of this using at and cron.

Mysql Connection - This is done via MySQLdb add-on.

Upvotes: 1

dlamblin
dlamblin

Reputation: 45381

Tools languages infrastructure? Vim, bash, cron. Unless I misunderstand your question. For MySQL type tasks I use perl or python.

Upvotes: 1

Jim Mitchener
Jim Mitchener

Reputation: 9003

Bash scripts are still used for almost everything. It really depends on what the job is at hand. I tend to resort to Perl for heavy text processing with its wonderful regex support and short syntax for instant hackability. I might use Python if I want to do something more complex that will need to be maintained over time. It really depends. But for the majority of "batch scripting" Bash scripts are used.

Upvotes: 1

richo
richo

Reputation: 8999

cron jobs are what you're looking for.

Upvotes: 1

Related Questions