Jump to content

This topic is 2061 days old. Please don't post here. Open a new topic instead.

Recommended Posts

Okay… here’s a toughy. I’ve got a corrupt file, so we need to get all the data into a clone of a known-good backup. Obviously I can import it directly, but to be absolutely certain we’re getting rid of all corruption, my supervisor has requested that I export the data to merge files and then re-import it into the known-good clone, which sounds great to me. Unfortunately, there’s a snag: a lot of the data includes formatted text, which is very important to maintain. So, here’s the solution I came up with, repeated for each table in the solution:

 

1) Export all modification information (mod account, mod timestamp, etc.) to a merge file.

 

2) Loop through each field of each record of the current table and check for formatted text, using the following calculation:

GetAsCSS ( GetField ( $FieldName ) ) ≠
GetAsCSS ( TextFormatRemove ( GetField ( $FieldName ) ) )
Any data that returns True is replaced with its CSS equivalent.
 
3) Export the data to a second merge file.
 
4) Import the data from the second merge file into the backup.
 
5) Perform step 2 in reverse, using a slightly modified version of Christopher Gauntt’s CSStoFMText custom function.
 
6) Import the modification info from the first merge file into the backup.
 
 
I’ve gotten the script to the point where it’s working perfectly, but it’s taking forever: it’s been running step 2 on a particular table (62 fields, 2.3 million records) for over 12 hours and is still only about 60% complete. Once it’s done, it still has to run the remaining four steps on that table (I expect step 5 to take as long), then process the remaining 33 tables (most of which are smaller, but still).
 
Any thoughts on what I can do to optimize this further? Perhaps some method of finding formatted text?
 
Thanks!
Link to post
Share on other sites

Forgive me if this wouldn't work for some obvious reason I'm not seeing, but have you considered exporting to a FileMaker file rather than text?

Link to post
Share on other sites

This topic is 2061 days old. Please don't post here. Open a new topic instead.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Similar Content

    • By sal88
      Hi all
      I have a customer list in a drop down. The drop down field has an On Save script trigger which then performs a find in the same layout for the selected customer. The drop down field is configured to 'go to next object using' return.
      There is also an On Record Load trigger on the layout which pops up with a dialog if the customer's Attention field is populated.
      This all works fine unless I utilise the option to save the drop down field with the enter button. It ends up looking like this, with the white background. Once the user clicks OK the layout 'appears'.

      Thanks!
       
       
       
    • By HOnza
      Excerpts from my “FileMaker 17 Performance Lab” session are available, covering some of the topics not covered anywhere else, as well as from my presentation of upgraded demo integrating 11 hardware technologies. Take a look at highlights of the FileMaker Konferenz 2018 and learn how field dependencies can slow down your Set Field, how to minimize the impact of slow network connection, why to avoid overusing conditional formatting, how to benchmark and optimize layout rendering, or how to integrate RFID, NFC, mobile printer, motors, digital scale, and other hardware with FileMaker app running on iOS.
      Click here for the whole article and videos
       




    • By HOnza
      [FOR IMMEDIATE RELEASE]
       
      24U releases 24U Toolbox Plug-In 3.0 for FileMaker 17
      The popular toolbox of external functions for FileMaker® 17 now can convert rich-text to HTML with CSS
       
      Prague, CZ — October 30, 2018 — 24U Software has released version 3.0 of 24U Toolbox Plug-In, and update of their popular toolbox of useful external function for daily use in FileMaker Pro 13 thru 17
       
      The new version adds convert rich-text to HTML with CSS, support for Native iOS FM Go apps and Support for FileMaker Cloud.
      24U Toolbox Plug-In 3 can convert rich-text to HTML with CSS, producing much cleared and customizable results than GetAsCSS. By producing clean dynamic HTML content the websites will render faster and rank higher in search engines.
       
      What's new in this version:
      - Get rich text as HTML to generate crystal clean HTML and CSS
      - Support for Native iOS FM Go apps
      - Support for FileMaker Cloud
       
      About 24U Toolbox Plug-In:
      24U Toolbox Plug-In is a robust FileMaker Pro plug-in which allows your FileMaker Pro databases to:
      - Execute FileMaker or System shell scripts to automate tasks inside/outside of FileMaker Pro
      - Execute SQL commands to develop faster and keep your solution clean
      - Match & replace regular expressions to find & replace text fast using patterns
      - Merge data values into text while maintaining styling to apply templates quickly and easily
      - Lookup DNS name for IP & vice versa to find out who is connecting from where
      - Get current public IP to know more about clients and discover potential security attacks
      - Define global keyboard shortcuts to avoid unnecessarily excessive use of script triggers
      - Obtain precise microsecond timestamps to measure nearly unmeasurable chunks of time
      - Get the type of a FileMaker variable to let your scripts make the right decisions
      - Share variables between databases to avoid creating too many external data sources
       
      Recommended configuration:
      Mac: Any Intel Mac, 4 GB RAM, Mac OS X 10.10 Yosemite, FileMaker Pro 16 Advanced or newer.
      Windows: 1.6 GHz processor, 4 GB RAM, Windows 8.1 or newer, FileMaker Pro 16 Advanced or newer.
      FileMaker Cloud: EC2 Instance type: t2.medium, FileMaker Cloud 17 or newer.
       
      Minimum configuration:
      Mac: Any Intel Mac since 2008, 2048 MB RAM, Mac OS X 10.10 Yosemite, FileMaker Pro 13.
      Windows: 1 GHz processor, 2048 MB RAM, Windows 7, FileMaker Pro 13.
      FileMaker Cloud: EC2 Instance type: t2.small, FileMaker Cloud 16.
       
      Availability and Pricing of  24U Toolbox Plug-In 3.0
      24U Toolbox Plug-In 3.0 is immediately available for download free of charge as a fully functional 14-days trial version, which can be activated after purchasing a license.
      Licenses for 24U SimpleHasp Plug-In start at US$49 per user, $499 per server. Volume discounts are available for 5 or more users.
      Existing customers can upgrade to the new version at 30 % off the regular prices. Until 13 November 2018 customers can upgrade at 50 % off the regular prices!
       
      More information: https://www.24usoftware.com/Toolbox/
      Download: https://www.24uSoftware.com/Toolbox#download
      Buy: https://www.24uSoftware.com/Toolbox#buy
       
      About 24U Software
      With a team of full-time testers, web, plug-in, Mac, Windows, Linux, iOS, and Android native app developers, and FileMaker certified developers, led by the FileMaker DevCon 2015 Developer Cup champion, 24U excels in taking care of existing FileMaker solutions, optimizing their performance, identifying and resolving potential issues and liabilities, improving reliability, stability and scalability, integrating them with other systems including various hardware devices, and extending them beyond expectation.
       
      As a platinum member of the FileMaker Business Alliance, we help global businesses around the world to maintain sustainable growth by working with their in-house developers or completely taking care of the maintenance and development of their business solutions.
       
      Customer contact:
      HOnza Koudelka
      Software Division Manager, 24U s.r.o.
      sales@24usoftware.com
      http://www.24usoftware.com
       
      Press Contact:
      pr@24uSoftware.com
       
      ###
       
      (c)2018 24U s.r.o. All rights reserved. 24U, 24U Software, and FM Bench are trademarks of 24U s.r.o. FileMaker is a trademark of FileMaker, Inc., registered in the U.S. and other countries. All other trademarks are the property of their respective owners. Pricing and availability are subject to change without notice.
    • By Matt Navarre
      Announcing fmSearchResults 5. This latest download delivers new features with simple implementation. And it's free!
      fmSearchResults 5 adds fast, multi-table searching to your FileMaker solutions by importing a few scripts and pasting a simple search field on to your layouts. It feels like the type of Google search that all your users are already familiar with, and it’s far more powerful than FileMaker’s Quickfind feature, because it searches across multiple tables and has data-type awareness.
      Read more about the new features and download it now!
      Implementation is so simple, watch me do it 5 minutes (and 12 seconds).
    • By brokenmouse
      I have a database that we use to update our website inventory. A few years ago we began offering customized merchandise that gets dropshipped direct from suppliers.  Suppliers give us data feed files with their inventory levels, pricing, etc. and this file manipulates the data. It takes our current web database, compares values and exports those products with their updated values.
      Importing the data and exporting used to take less than hour but now it takes several since the size of the web database has grown and the number of suppliers has grown.  Everything is automated through scripts. In the main table (web database), the proposed quantities, pricing, leadtime, variations, etc. all use unstored calculated fields to determine the new value. I then have a separate field which is used to flag items that need updating. The major bottleneck of the entire process is the searching of this field. It can take sometimes over an hour to search this field. Other steps like exporting the changes can take a while, too.
      I have done some things to optimize the database but it still seems that these unstored calc fields are what is dragging everything down. I have tried replacing some of those calc fields with text/num fields with "replace field contents" script steps (or auto entry) but it does not seem to make a difference because of the indexing.  The database is not hosted or shared and my computer has decent specs with an SSD HD. I've got a simplified design chart attached for reference.
      I am not sure that this is what comes with a large complex database file or if my design is flawed. The only two things I can think to try to reduce the processing time is:
      1) Rewrite the scripts to update the supplier/inventory table records instead of replacing the records fresh each time.
      2) Use a looped set field script to set the "change flag" field and/or the other updated price/qty/etc fields
       
      Any thoughts or advice is much appreciated.

  • Who Viewed the Topic

    2 members have viewed this topic:
    HOnza  ibobo 

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.