Have not trolled the source enough to know who specifically to thank, so a general thank you to whoever is responsible for \e(f) in psql and the command DO. I have been working on a fairly complex plpythonu function in a 9.0 database and the combination of the afore mentioned tools made the development much easier. The ability to test Python code snippets by writing DO functions inside an editor inside psql is priceless.
Author Archives: aklaver
PostgreSQL 9.1, first beta around the corner
Joshua Berkus recently gave a heads up on the Advocacy list of the forthcoming release of the first beta for PostgreSQL 9.1, tentatively scheduled for May 3. I have been following the development fairly closely, though given the number of changes, not in depth. Good time to point out the Waiting for.. blog series from Hubert ‘depesz’ Lubaczewski found here. It covers many of the changes and uses examples to illustrate, a very handy resource. So what is 9.1 offering?
First some of the headliners.
Synchronous Replication:
Extends the built in replication to eliminate any lag between the master and standby databases. For those who absolutely need a 1:1 correspondence with their standbys.
Per column collation:
A different language for each column, the Tower of Babel realized:)
Unlogged tables:
Permission is granted to run with scissors. You can now create tables that do not log to the Write Ahead Log(WAL). This makes data operations on the table faster, at the cost of data security. Data will not be recovered in the event of a crash, nor are unlogged tables replicated.
SE-Postgres.
Speaking of security Postgres will now have access to the SE-Linux framework.
The above are great and I will be exploring them at some point, but what follows are the new features I am really excited about.
PL/Python changes:
Thanks to Jan Urbanski and others there have been a host of changes to the procedural language support for Python. To mention a few; table function support, better error handling, explicit subtransactions.
pg_basebackup:
Thanks Magnus. A way of doing a basebackup for replication using a Postgres connection. Very handy. Along with this, the replication permission for non-superusers.
SQL/MED:
Basically pulling external data sources into Postgres. The implementation for 9.1 is basic, but the potential is too great to ignore.
Extensions:
A easier way to work with database plugins. Combined with the PGXN site a definite plus.
I have just scratched the surface on what is new and improved in PostgreSQL 9.1, for a more in depth look I suggest the latest alpha Release Notes. I mentioned specific people above because I have associated their names with particular features. Lest I forget, a big thank you to all those who have participated. There is much to be proud of in this release as in the previous ones.
Troubleshooting
Some thoughts on the art and science of troubleshooting, or how not to resort to the bigger hammer. What I offer up is based on my experience trying to solve problems across an array of disciplines, from truck repair to software debugging. There are common elements to all that are listed below.
1) Work from the known to the unknown not the other way.
This is best illustrated by working with electrical circuits. There are any number of reasons why electricity is not reaching the end of circuit and working backwards through the circuit often leads down false branches. It is a lot easier to start at the closest panel box and follow the circuit out until voltage is lost.
2) Keep notes.
I am not talking an encyclopedic compendium, something more along the likes of a bread crumb trail to keep from going in circles or trodding down old paths. To extend the electrical example, was that the green wire with white trace or white wire with green trace?
3) All information is important.
I will not recount the number of times I ignored an error message because I ‘knew’ it was not important, much to my later chagrin.
4) KISS(Keep It Simple Stupid).
The poor mans version of Occam’s Razor. If the choice is between an alien gamma ray device flipping bits in the computer or poor quality memory chips, go with the latter.
5) First guess.
This is the you know more than you think principle. Also known as listening to the little voice. In the absence of definite plan, go with the first thought that strikes you. I use to spend a lot of time fighting the little voice, until I realized it was the part of me that was actually paying attention.
6) Walk away.
My dog has helped me solve many problems. I don’t know how many times I took a break to walk the dog and had the answer present itself while the dog was leaving a message. Disconnecting from the problem is often good way to make the connection.
SQL inside Jedit
I use Jedit as my primary text editing tool. I recently ran across a plugin (SQL) to the editor that allows for running SQL from within an editor buffer. What follows is an explanation of how to set it up and run it. Some background, Jedit is a Java application so if you plan to use it you will need a JVM installed. I have found that to have it run well you need to use the Sun(Oracle) Java code.
Set up:
Java runtime 1.6+
Jedit 4.3+
Installing the SQL plugin from the Plugin Manager will install dependencies most notable Project Viewer.
The set up for SQL has global and per project parts.
The global portion is set up through the Plugin Manager plugin options menu. By default SQL only returns 10 rows and that can be changed here. I have yet to find a way to get it to return with no limits. I would also suggest checking the Show Toolbar and Popup messages options. The most important global setting is the JDBC page, where you set up the path to the JDBC drivers you wish to use. SQL supports quite a few databases (Oracle, MySQL, PostgreSQL, DB2, Sybase, Microsoft SQL Server see help file), though notably not SQLite.
Project Viewer (PV) is installed as a dependency because it is used to create per project settings for SQL. It is possible to use a PV project for just the SQL settings and not use the rest of the package, though I would suggest checking PV out. One note, PV seems to like to throw non fatal errors on a regular basis. They seem to be related to redraw operations and so far have not caused me any problems. Once you have created a project you can go to the project SQL setting to enter the necessary information. See screenshot:
Once you have the server set up the fun starts. As an example I wrote a script to CREATE a table and another to populate it with data. If you look at the screenshot you will see the SQL toolbar I alluded to earlier, it starts with Database:. The second button to the right of the drop down box is the Execute buffer command. Clicking on that causes the entire sql script in the selected buffer to execute. The result is shown in the message box, in this case for the CREATE TABLE statement. I did this again on the buffer holding the INSERT commands .
So now we have a table with data, now what? See below. In the right buffer I have written a SELECT statement against the previously created table. In this case I could have done Execute buffer but for illustration used another method Execute selection. This is the first button to the right of the drop down list. It only runs whatever text is selected.
So what about variable substitution? In the first screenshot below I have used the drop down list Preprocessors to select variable substitution. In the left buffer is another SELECT query this time using the substitution variable ‘?’. The second screenshot shows the dialog box that pops up for data entry. For more than one variable the plugin will cycle through them and present a dialog for each.
One more interesting feature, SqlVFS . This allows one to browse the database through the Jedit File Browser. When you open the File Browser select the Plugins dropdown and then Show databases. Double clicking on the database allows you start drilling down through the database structure.
In the short time I have been using this tool I have found it quite helpful, so I thought would pass along my experience. Saves a round trip between the text editor and a terminal window.
PostgreSQL Roles
Today I got involved in a discussion on the General mailing list about using ROLE in Postgres. I realized this is a topic that can get confusing in a hurry and I thought I might try to clear things up, in my mind if nothing else. Part of the confusion arises from a historical carry over. In versions of Postgres prior to 8.1 there where USER and GROUP. (An aside, I found this out by using this : Feature Matrix, a very handy tool. ) These where distinct and operated somewhat like their OS equivalents, in that users logged in and then could be part of a group. In 8.1+, in other words all current community supported versions, USER and GROUP have been replaced by ROLE. The confusion arises because while the current underlying mechanism is based solely on ROLE, the USER and GROUP commands have been maintained for backwards compatibility. So when you use the SQL command CREATE USER in 8.1+ you are just creating a ROLE with the clause LOGIN whereas CREATE GROUP is just CREATE ROLE which by default has the clause NOLOGIN. For my own personal sanity I just use the CREATE ROLE command and pretend USER and GROUP do not exist. Since we already done the rabbit hole I might as well mention the psql \d commands that apply as well. Again for historical reasons these are \du and \dg which at one time listed users and groups respectively. Now they both return the same thing, a list of the roles. The thing to remember is that unless you are working with a pre 8.1 database you are working with some variation of a ROLE.
So what does a ROLE get you? As was implied above a ROLE can have the characteristics of a user or group. For my own use I follow the pattern used by the current CREATE USER/GROUP commands where a ROLE with LOGIN set is a taken to be a user and one with NOLOGIN is equivalent to a group. Note: This is just a convenient way of viewing things and you will not see a ROLE defined that way when you list them. Assuming that pattern then it possible to create group like ROLES and then assign user like ROLES membership in them. For instance:
‘Groups’:
CREATE ROLE accounting;
CREATE ROLE sales;
CREATE ROLE production;
‘Users’
CREATE ROLE aklaver LOGIN IN ROLE accounting,sales;
CREATE ROLE daffy_duck LOGIN IN ROLE production;
If you want to change role membership after CREATE ROLE then you will need to use the GRANT or REVOKE commands.
Where this pays off is in object permissions. Give the ‘group’ roles permissions to their respective objects and then add or drop ‘users’ to the ‘groups’ as needed. This way you can maintain access flexibility without continually having to change permissions on objects.
I have touched on just the bare basics, this is a subject that covers a lot of ground. To really learn it requires multiple passes through the documentation and plenty of hands on testing.
Quantas gripe sheet
Ran across this years back, it still gives me a chuckle:
After every flight, Qantas pilots fill out a form, called a “gripe sheet,” which tells mechanics about the problems with the aircraft.
The mechanics correct the problems, document their repairs on the form, and then pilots review the gripe sheets before the next flight. Never let it be said that ground crews lack a sense of humor.
Here are some actual maintenance complaints submitted by Qantas’ pilots (marked with a “P”) and the solutions recorded (marked with a “S”) by maintenance engineers.
By the way, Qantas is the only major airline that has never had an accident.
P: Left inside main tire almost needs replacement.
S: Almost replaced left inside main tire.
P: Test flight OK, except auto-land very rough.
S: Auto-land not installed on this aircraft.
P: Something loose in cockpit.
S: Something tightening in cockpit.
P: Dead bugs on windshield.
S: Live bugs on back-order.
P: Autopilot in altitude-hold mode produces a 200 feet per minute descent.
S: Cannot reproduce problem on ground.
P: Evidence of leak on right main landing gear.
S: Evidence removed.
P: DME volume unbelievably loud.
S: DME volume set to more believable level.
P: Friction locks cause throttle levers to stick.
S: That’s what they’re for.
P: IFF inoperative.
S: IFF always inoperative in OFF mode.
P: Suspected crack in windshield.
S: Suspect you’re right.
P: Number 3 engine missing.
S: Engine found on right wing after brief search.
P: Aircraft handles funny.
S: Aircraft warned to straighten up, fly right, and be serious.
P: Target radar hums.
S: Reprogramming target radar with lyrics.
P: Mouse in cockpit.
S: Cat installed.
And the best one for last…
P: Noise coming from under instrument panel. Sounds like a midget pounding on something with a hammer.
S: Took hammer away from midget.
First Postgres post
This is more of a test post than anything. A way of checking that I have my feed to Planet PostgreSQL set up right. The point being to have only Postgres related posts come through, not my thoughts on cat juggling(thanks Steve Martin) or other such topics. Should the system work I will follow with a post on Streaming replication/Hot Standby/WAL archiving. As Bruce Momjian is aware I have been experimenting with SR. That has led me to trying to figure out the relationships between the previously mentioned components. More thoughts on that later.
Life Rule 2
“Easy is difficult.”
Subtitled the tale of two diesels. The principals:
Woods & Copland 1942 Cat
The machines pictured are two I had personal experience with. So what do a couple of pieces of heavy equipment have to do with the rule? In this case starting the beasts . The old Cat had a multiple stage means of starting. The primary motor was a diesel, but to start it you first had to deal with a gas ‘pony’ motor. A rough outline; crank the pony motor, let it warm up, use hand clutch to have pony motor spin diesel, spin diesel for awhile, diesel coughs to life, let diesel warm up, do work. For a more detailed account see (http://www.vannattabros.com/dozer3.html) and read the part ‘Starting an Old Cat’. Compare this to starting the Woods & Copland tractor; turn key . From the end user view point starting the tractor is easy compared to turning over the Cat. To make it easy though was difficult. The reason the Cat had the pony motor was that the electrical starter motor/battery combination’s of the day were not strong enough to spin the diesel motor. Over the years much work was done on this to get to the system that was on the tractor. Even so this particular tractor had a wrinkle. The earlier models from Woods and Copland used air starters to spin the 320 HP motor because 12V starters where not powerful/reliable enough. To make an electrical start system work this model actually had a 24V starter. The wrinkle was that the rest of the electrical system was 12V. How did this work you ask? Enter a electronic switch over module. The tractor had two 12V batteries. When you turned the key to the start position it switched the batteries into a series connection effectively creating a 24V battery. It also isolated the rest of the electrical system from the 24V. When you let the key return from the start position it put the batteries back into a parallel connection and reconnected the non-starter electrical system. Why do all this? 24V components are a lot more expensive then 12V ones and harder to find. This method was a compromise between power when you needed it and economy elsewhere. Getting back to the rule. There is easy and there is easy. Many times what is presented as easy it really a veneer over a difficult process, think usability studies. Most I have seen are about hiding the complexity from the end user. Said end user hopefully comes away saying ‘that was easy’. What should not be lost sight of though is that the underlying process is still difficult. Hence tech support.
Life Rule 1
Part of me wanted to do random(life_rules), but then the Dutchman in me kicked in. So I will start at the beginning.
“The good will take care of itself, its the bad you have to plan for. ”
Nothing too original, this is the basic premise of insurance. It is one of those things that people, myself included, often pay lip service to without really following through on. The importance of the follow through has more to do with state of mind than anything. Too many times I have seen manageable problems descend into crises solely because the people involved never entertained the thought that something could go wrong. The act of planning prepares you for the unfortunate. Even if you do not have a fully fleshed out plan, the mere act of anticipating the need is often sufficient to avert a situation from spiraling out of control. An example from my greenhouse days, no matter the amount of preventative maintenance you do, a boiler will choose 3 AM on the coldest night of the year to take a break. A boiler is the intersection of several systems; solid state controls, electrical wiring, vent plumbing, water plumbing and gas plumbing. Further, modern boilers are shrines to safety and have more interlocks between these systems than you can imagine. Coming up with a plan that deals with every eventuality is possible but ultimately counter productive. Too much time spent studying the plan, not enough solving the problem. The key is to have a skeleton troubleshooting scheme either in your mind or on paper, preferably both. To complement this, sufficient tools and materials. I am using an expansive definition of tools to include written matter, such as control schematics. Materials would be common replacement parts i.e. flame sensor,thermocouple, relays,etc. With the above in hand it is possible to bring a boiler back to life under trying circumstances. Truth compels me to also acknowledge that when ever possible I tried to hedge my bets by having redundant heat sources available. Nothing like having a couple of unit heaters cranking along to lower the blood pressure a few notches. All part of the planning process. It is easy to get overly pessimistic if you carry this rule to the extreme. As in many things moderation is in order. Where that balance point is, comes with time in the School of Hard Knocks. Taking a long walk down a country road because you had a flat tire, a spare tire and no lug wrench is a tremendous learning experience.
Starting out
My attempt at extended thought on Facebook ran into a character limit for status messages. I have thought about cranking up a Blog for some time now; the Facebook limit has motivated me to start. Per the title of this site the entries will cover a spectrum of topics. To get a sense of where the ideas come from, some background. I am at heart a scientist. I have a BS in Fisheries Science and an MS in Environmental Science and my father is Geologist. Science is what I have known since I can remember. That being said, it is not what I do now. At this point in time I am a duly licensed General Contractor in the state of Washington. The reality is that I am a glorified handy man. The path to this has been anything but straight. In roughly chronological order the stops have been– grocery store stocker, catfish farmer, crop duster loader, monument(headstones) installer, greenhouse worker, delivery driver, greenhouse maintenance supervisor, landscaper, retail nursery sales, budding software developer, self employed contractor. Basically a testament to a restless mind. The good is that I have learned a lot, the bad is it involved starting over many times. There are many lessons and stories to tell and over the course of time I will cover them one way or another. For now though I will stick to what I call Adrians Rules of Life. There are five and like most things they are not necessarily original to me. They do however seem to encapsulate my experiences. With out further ado the rules, further detail to follow:
1) The good will take care of itself, its the bad you have to plan for.
2) Easy is difficult.
3) Its all the same problem.
4) The more I know the less I know.
5) Don’t depend on a negative.