Monday, December 15, 2008

New FamilySearch

Remember, I am not a spokesperson for either Ancestry.com or FamilySearch International. This article gives my opinion as to the history, current state, and future of New FamilySearch. I have not included any confidential information and I am solely responsible for the contents of this article.

Dear Ancestry Insider,

I have a question and didn't want to look as uninformed as I really am by posting online. I ask for your patience, please. I've not been deeply involved with genealogy for about 6 years, but just got asked to teach a class in church at BYU, so I am frantically trying to get back up to speed.

I thought that there were temple districts using NFS as a beta test. Sacramento is one that I know for sure is. Aren't there several districts already using NFS in an effort to get all the kinks worked out? I read your announcement about Las Vegas and began to wonder. How will their use be different from the temple districts that are just testing it? And what are the implications for the rest of us?

Signed,
Ollie Magneson

Dear Ollie,

Don't worry; I wouldn't think of disclosing your uninformed-state secret. (Hold it, everyone! Before you send me nasty messages telling me I'm a discredit to my employer, I changed Mark's name... Oh, dang...)

Seriously, here's the skinny. New FamilySearch (NFS) is a single family tree that all of us share and work on in common, as if we all shared one PAF file. FamilySearch is temporarily calling it New FamilySearch and it is temporarily located at http://new.familysearch.org. But eventually it will be called FamilySearch Family Tree and will be relocated to www.familysearch.org. The 1.0 version will be available to all, members of the Church and non-members alike.

However, the first priority for NFS was to stop the flood of duplicate temple ordinances by replacing TempleReady and the associated need to check the International Genealogical Index (IGI) to avoid duplication. Accordingly, a version 0.9 was written to this end. It lacks many niceties considered standard in a genealogy product, but those features are not central to replacing TempleReady and can wait for a 1.0 product.

After alpha and beta testing were complete, a multiple-phase rollout was commenced on 26 June 2007 when St. Louis started using NFS. From that moment on, NFS has not been in some extended beta test as some suppose, but has been in real use in real temples.

Early users of NFS found bugs, of course, as well as user interface problems. That is one reason for doing a phased release. But these problems were nothing, in retrospect. Like the hero of a tragedy, NFS 0.9 contained an unknown fatal flaw that doomed it to failure as soon as the rollout began. Ironically, the flaw arises out of the problem that NFS is designed to avoid: too much duplication.

Our hero's tragic flaw

Somewhere along the line, two conflicting mantras were established for NFS. Our hero's fatal flaw results from an unforeseen interaction between the two.

  1. No one can change your data except you.
  2. To keep things simple, New FamilySearch combines Ancestral File, Pedigree Resource File and the International Genealogy Index into one database.

Not knowing the number of duplicates beforehand, the first mantra was kept by keeping a full copy of every piece of data, no matter how often duplicated. After the rollout began, users began combining duplicates from these three databases. Some individuals were duplicated many, many more times than expected and when NFS users combined them, "individuals of unusual size" (IOUS) started to grow.

Steps were taken to slow IOUS growth. The addition of PRF disks was halted. The import of complete GEDCOMs was frowned upon. And NFS continued to perform its primary function, adequately replacing TempleReady.

The Arizona

Then on 5 February 2008 the rollout hit Arizona. Because of the large number of Church members there who are descended from early Church members, the growth rate of IOUSes exploded and IOUSes became large enough to swamp the computer servers running NFS. The system was sometimes too slow to use. I'm sure within a week FamilySearch knew they had big problems. The decision to freeze would have been gut wrenching and probably had to reach to the highest levels. On the 19th, word first leaked out. On the 21st, official word was sent out. The rollout was stopped, frozen solid.

Emergency steps were taken to rehabilitate system performance. Hard limits were placed on the number of duplicates that can be combined. (I believe it is currently 85?) GEDCOM import size was restricted. I imagine the length of the delay was predicated on how long it took database engineers to scan through all the millions of individuals in NFS to find and split the IOUSes into pieces small enough for the system to handle.This had to be done while the system was in operation, actively serving 26 temples.

The result was an NFS that worked near flawlessly as a TempleReady replacement for anyone who doesn't have ancestors that were famous or were members of the Church. These ancestors were the ones becoming IOUSes. When the freeze thawed, the rollout could continue only in districts where most members didn't have many ancestors fitting this characterization. By 14 October the tragedy had run its course. Utah, Idaho and Las Vegas have been on hold ever since, waiting for a true fix to the IOUS problem. Rumors have pointed to Q3 or Q4 of next year before this newer than New FamilySearch will be ready.

Family Tree

While all this was going on, work on a 1.0 user interface was progressing. Developers are using a system called Agile Development that encourages regular user interaction during iterative development. This allows us, the future users of the program, to try things out along the way, identify design flaws and influence the product before it is set in stone. If you have a New FamilySearch account, you should feel a responsibility to do this at http://labs.familysearch.org.

That brings you up to the unexpected announcement that Vegas was going live without the rest of us! What does that mean? Obviously, FamilySearch feels like the system is robust enough to handle the additional traffic and that letting Las Vegas start using the system is worthwhile, despite members inability to combine all duplicates together.

Teaching Church Members in Utah

In the mean time, what do you teach in Utah?

  • Teach the members the fundamentals of source-based research and analysis. It's the only way the quagmire of NFS will ever get cleaned up.
  • Teach the members the extreme importance of avoiding further duplication. Each hour of necessary research now will save ten later on.
  • Teach the members to follow the rules for temple submissions. Stop submitting people you're not related to. Stop submitting people you don't have nearest-kin permission for. Those that do so are embarrassing the Church and turning away the hearts of those we'd like to bless. You will be held responsible for those who might have accepted ordinances (alive or dead), but didn't because of your disobedience.
  • Teach members the importance of turning their hearts in two directions. Those with Church-member ancestors should seek out their spiritual experiences, histories and photographs. Teach all members to never let their own spiritual experience go unrecorded, for their future posterity.
  • Teach members not to feel guilty. We don't need everyone to do everything; we just hope everyone can do something. Do only what you have time to do. FamilySearch Indexing is great for students who only have 10 minutes here or there for family history. Just be sure to Save to Server, so your few minutes of work can be passed on to another to finish.

I hope this helps. Above all, have fun! This is a glorious work.

Sincerely,

The Ancestry Insider

3 comments:

  1. One of the questions asked was about which Temple districts are using newFamilySearch. The answer to that question is that all Temple districts outside of Utah and Idaho except for a few in the Orient with complex alphabets and Las Vegas Nevada are currently using newFamilySearch to clear names for the Temple.

    What our family history center is concentrating our training on is new research. For many years most members spend most of their time around discovering what has already been done rather that expanding research to new lines. Most members spend their family history time gathering records from Ancestral File, the IGI, and the Pedigree Resource File and constructing from that data their family trees, they then proceed to check for missing ordinances. This has been done so many times that many individuals have had ordinances repeated many many times. Our centers focus is on finding ancestors or decedents of ancestors who are not already in those databases. That is the future of Family History Centers, assisting in new Research.

    The FamilySearch FamilyTree program will assist us in that effort by providing a place where all of the completed work will be available for anyone to view online in pedigrees without the need for each person to reconstruct that information and check for missing ordinances. You will then be able to see where ancestor lines end and where decedent research needs to be done.

    ReplyDelete
  2. Dear AI, I'm not a Church member nor much of a FamilySearch International user (just got introduced to genealogy and your blog through ancestry.com a year or two ago) but am struck by this sentence in your current post:

    "New FamilySearch (NFS) is a single family tree that all of us share and work on in common, as if we all shared one PAF file."

    This almost sounds like it could cause OneWorldTree types of problems. Curious as to how NFS guards against similar types of problems?

    Certainly would appreciate any insight you can provide on this.

    ReplyDelete
  3. datadilettante, FamilySearch had similar problems that other large Tree type programs experience when they created the Ancestral File which is a merged family tree of many many families. This new concept which is being called FamilySearch FamilyTree creates a folder for each individual then links these folders into pedigrees. Then all of the records for an individual are placed into the folders. No information is merged and misfiled records can be removed from folders. This solves the problems other large trees experience by merging multiple records into one record.

    ReplyDelete

Note: Only a member of this blog may post a comment.