Powershell Script to Download Documents from Document Library

Follow a simple Powershell script to download all documents from a document library. It can be neat when you have large document libraries and don’t want to use Explorer View to transfer files.

 

$destination = “C:\\temp\\”
$web = Get-SPWeb -Identity “http://%5BSERVER NAME]/[SP WEB]”

$folderURL = “[DOCUMENT LIBRARY]/”

function ProcessFolder {
param($folderUrl)
$folder = $web.GetFolder($folderUrl)
foreach ($file in $folder.Files) {
#Ensure destination directory
$destinationfolder = $destination + “/” + $folder.Url
if (!(Test-Path -path $destinationfolder))
{
$dest = New-Item $destinationfolder -type directory
}
#Download file
$binary = $file.OpenBinary()
$stream = New-Object System.IO.FileStream($destinationfolder + “/” + $file.Name), Create
$writer = New-Object System.IO.BinaryWriter($stream)
$writer.write($binary)
$writer.Close()
}

foreach ($subfolder in $folder.SubFolders)
{
write-output $subfolder.Url
ProcessFolder($subfolder.Url)
}
}

ProcessFolder($folderURL)

 

See you,
Amadeu.

Code to download files from a SharePoint 2010 site using Claims Based Authentication

Trying to download some images from a SharePoint 2010 Image Library using code I was getting HPPT Status 401 – Unauthorized.

First thing I check was the user name and password in a browser window. It worked just fine.

This code used to work for SharePoint 2007 but then I recalled the site uses Claims Based Authentication and it needs a little change in order to authenticate a user on 2010.

Checking the definitions for Windows Authentication requests on SharePoint 2010 I saw we need to specify the X-FORMS_BASED_AUTH_ACCEPTED and the user agent HTTP headers .

Follow a code snippet:

string url = @"File URL";
string userName = "account";
string password = "password";
string domain = "domain";


NetworkCredential credential = new NetworkCredential(userName, password, domain);
WebClient client = new WebClient();
client.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f");
client.Headers.Add(HttpRequestHeader.UserAgent, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
client.Credentials = credential;
byte[] data = client.DownloadData(url);


After this code change it worked just fine.

See you,

Amadeu.

HTTPS Connection Closed and Status is 200

Testing an IIS web site using SSL certificates I was getting an error on my browser. Using Fiddler I could see I was getting HTTP status 200 but no response on my site.

On the response headers section for the HTTP request on Fiddler I could see the connection had been established and then on the transport info I could see it had been closed.

At first I thought it could be some firewall rule or network issue. Checking the web site configuration on IIS, I opened the bindings for the web site and figured out the site web was not configured to use a SSL Certificate even tough it was using port 443 and HTTPS protocol. That was the reason the HTTP connection was being closed, IIS couldn’t find a certificate to sign the response to the browser.

After selecting the appropriate SSL certificate for the web site, it worked fine.

 

See you,

Amadeu.

FAST Impulse – User name or password is incorrect on BizMan

Out of a sudden we started getting a message about user names and passwords being wrong on BizMan in a FAST ESP + Impulse environment.

The users were trying to access the BizMan which is the Impulse tool to create and manage search profiles, facets, promotions, search terms and search result boosts.

The error message was “incorrect user name or password“. This lead us to think somebody changed his password and didn’t remember it. But then we checked all the users we knew were having the same issue and seeing same error message. Suspicious, isn’t it?

So we checked the Impulse BizMan database and, believe you or not, on the ImpulseBizMan.users table we could find all the passwords stored as clear text. We checked the passwords on the table and they all matched the ones we used to try to log in to BizMan.

I checked the error messages on all log files on the ESP\var\logs\adminserver folder and found no error messages about user’s logging process.

Since BizMan service/web site depends on the EML Server to get all the information from the Impulse BizMan database I toh. When I checked the logs for the EML Server I found lots of entries about an invalid character in one of the category ids in the category tree. The message was:

ERROR 2012-01-11 11:49:55,823 (no.fast.ecomm.taxonomy.dao.impl.UpdateTaxonomyTask:55) – The value of attribute “id” associated with an element type “category” must not contain the ‘<‘ character.
no.fast.esp.taxonomy.util.TaxonomyParserException: The value of attribute “id” associated with an element type “category” must not contain the ‘<‘ character.
at no.fast.esp.taxonomy.util.TaxonomyParser.readTaxonomy(TaxonomyParser.java:66)

I opened the category tree file and noticed it had been updated a few hours before. Good cue!

Reading the file we could find that on one of the categories we had a XML syntax error (missing a double quote):

<category id="categoryID><![CDATA[Category Name]]></category>

The correct syntax should be:

<category id="categoryID"><![CDATA[Category Name]]></category>

After fixing the file I had to restart the EML Server using the command:

nctrl restart emlserver

BizMan took a few minutes to reconnect to the EML Server and when it was done all users could log in successfully to BizMan.

As I buddy of mine said: “Seems they randomized error messages on the FAST to make it harder for you to debug”. I completely agree.

See you,

Amadeu.

The form cannot be rendered – SharePoint 2010 State Service

Testing a Publishing Portal on SharePoint 2010 I tried to submit a page thru a publishing workflow.

When I submitted it to approval I got the following error message:

The form cannot be rendered. This may be due to a misconfiguration of the Microsoft SharePoint Server State Service. For more information, contact your server administrator.

The solution to this error message can be found on this URL: http://www.topsharepoint.com/sharepoint-2010-state-service.

When we tried to apply the solution we ran the Farm Configuration Wizard on a pre-configured farm and broke our whole SharePoint farm.

Then I spent a bunch of hours to reconfigure the whole farm. So if I can advise you, don’t run the Farm Configuration Wizard, use the Powershell script to configure the State service instead.

It is pretty simple:

$svcApp = New-SPStateServiceApplication -Name "State Service"
New-SPStateServiceDatabase -Name "StateServiceDatabaseName" -ServiceApplication $svcApp
New-SPStateServiceApplicationProxy -Name "State Service" -ServiceApplication $svcApp -DefaultProxyGroup

See you,

Amadeu.

FAST Impulse – JDBC Connector Out of Memory Expection

We run a FAST ESP + Impulse implementation as our search platform. The main data load activities use the ELXT file format and the EXLT and JDBC Connectors to load documents into FAST index.

When we added a new content source to our loading process and created a new JDBC connector instance to load its data, we started getting an out of memory exception from the JDBC connector.

The error message I found on the connector’s log was:

Exception in thread “main” java.lang.OutOfMemoryError: Java heap space
at com.microsoft.sqlserver.jdbc.TDSPacket.<init>(Unknown Source)
at
com.microsoft.sqlserver.jdbc.TDSReader.readPacket(Unknown Source)
at
com.microsoft.sqlserver.jdbc.TDSReader.readPacket(Unknown Source)
at
com.microsoft.sqlserver.jdbc.TDSReader.readResponse(Unknown Source)
at
com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(Unknown
Source)
at
com.fastsearch.components.jdbcconnector.JdbcDocumentIterator.<init>(JdbcDocumentIterator.java:256)
at
com.fastsearch.components.jdbcconnector.JdbcAccess.iterator(JdbcAccess.java:224)
at
com.fastsearch.components.jdbcconnector.JdbcConnector.processDocuments(JdbcConnector.java:835)
at
com.fastsearch.components.jdbcconnector.JdbcConnector.runConnector(JdbcConnector.java:803)
at
com.fastsearch.components.jdbcconnector.JdbcConnector.main(JdbcConnector.java:1897)
Exception in thread “SubmitterThread” java.lang.OutOfMemoryError: Java
heap space
at
org.apache.commons.httpclient.ChunkedInputStream.exhaustInputStream(ChunkedInputStream.java:367)
at
org.apache.commons.httpclient.ContentLengthInputStream.close(ContentLengthInputStream.java:117)
at java.io.FilterInputStream.close(FilterInputStream.java:159)
at
org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java:176)
at
org.apache.commons.httpclient.AutoCloseInputStream.close(AutoCloseInputStream.java:140)
at
org.apache.commons.httpclient.HttpMethodBase.releaseConnection(HttpMethodBase.java:1078)
at
com.fastsearch.esp.content.http.SessionFactory.releaseConnection(SessionFactory.java:259)
at
com.fastsearch.esp.content.http.Session.processCall(Session.java:477)
at
com.fastsearch.esp.content.http.Session.process(Session.java:372)
at
com.fastsearch.esp.content.http.Dispatcher.send(Dispatcher.java:1104)
at
com.fastsearch.esp.content.http.ContentManager.submitContentOperations(ContentManager.java:247)
at
com.fastsearch.esp.content.http.ContentManager.submitContentOperations(ContentManager.java:206)
at
com.fastsearch.esp.content.feeding.DocumentSubmitter.doSubmitBatch(DocumentSubmitter.java:279)
at
com.fastsearch.esp.content.feeding.DocumentSubmitter.submitBatch(DocumentSubmitter.java:258)
at
com.fastsearch.esp.content.feeding.DocumentSubmitter.run(DocumentSubmitter.java:170)
at java.lang.Thread.run(Thread.java:595)
Full thread dump Java HotSpot(TM) Client VM (1.5.0_22-b03 mixed mode, sharing):

“DestroyJavaVM” prio=6 tid=0x00108510 nid=0xfb4 waiting on condition
[0x00000000..0x000bfab0]

“BatchTimerThread” prio=6 tid=0x03876588 nid=0x1b00 waiting on
condition [0x03e1f000..0x03e1fc30]
at java.lang.Thread.sleep(Native Method)
at
com.fastsearch.esp.content.http.BatchTimer.run(BatchTimer.java:32)
at java.lang.Thread.run(Thread.java:595)

“CallbackPollThread” prio=6 tid=0x035e33a8 nid=0xd24 in Object.wait()
[0x03bbf000..0x03bbfcb0]
at java.lang.Object.wait(Native Method)
– waiting on <0x24924490> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at
com.fastsearch.esp.content.http.Session.waitForActiveBatches(Session.java:292)
– locked <0x24924490> (a java.lang.Object)
at com.fastsearch.esp.content.http.Session.run(Session.java:244)
at java.lang.Thread.run(Thread.java:595)

“CallbackHandlerThread” prio=6 tid=0x033fa800 nid=0xec0 in
Object.wait() [0x03d9f000..0x03d9fd30]
at java.lang.Object.wait(Native Method)
– waiting on <0x249244f8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at
no.fast.util.SynchronizedQueue.dequeue(SynchronizedQueue.java:148)
– locked <0x249244f8> (a java.lang.Object)
at
no.fast.util.SynchronizedQueue.dequeue(SynchronizedQueue.java:97)
at
com.fastsearch.esp.content.http.CallbackHandler.run(CallbackHandler.java:145)
at java.lang.Thread.run(Thread.java:595)

“MultiThreadedHttpConnectionManager cleanup” daemon prio=6
tid=0x03610bb0 nid=0x1970 in Object.wait() [0x03c3f000..0x03c3fab0]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
– locked <0x2488d058> (a java.lang.ref.ReferenceQueue$Lock)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)

“Thread-0” daemon prio=6 tid=0x034daa98 nid=0xd34 waiting on condition
[0x03a2f000..0x03a2fbb0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)

“Low Memory Detector” daemon prio=6 tid=0x00f7ba68 nid=0x16bc runnable
[0x00000000..0x00000000]

“CompilerThread0” daemon prio=10 tid=0x00faaf38 nid=0x161c waiting on
condition [0x00000000..0x0326fa10]

“Signal Dispatcher” daemon prio=10 tid=0x00f7b0e0 nid=0xdc waiting on
condition [0x00000000..0x00000000]

“Finalizer” daemon prio=8 tid=0x00f78de8 nid=0x1688 in Object.wait()
[0x0316f000..0x0316fa30]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
– locked <0x24833758> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

“Reference Handler” daemon prio=10 tid=0x00f78240 nid=0x1978 in
Object.wait() [0x030ef000..0x030efab0]
at java.lang.Object.wait(Native Method)
– waiting on <0x248337e0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
– locked <0x248337e0> (a java.lang.ref.Reference$Lock)

“VM Thread” prio=10 tid=0x00fa2d48 nid=0x1ba0 runnable

“VM Periodic Task Thread” prio=10 tid=0x00f6af18 nid=0xefc waiting on condition

Reading the error message we could conclude the heap was running out of memory while processing the documents we were trying to load.

Checking the JAVA virtual machine configuration we found out the maximum memory for JAVA processes was set to 64 MB.

We changed it to 300 MB specifically for the JDBC connector.

In order to do it, we changed the JAVA options on the connector.windows.conf file under the FAST_ESP_Root\components\jdbcconnector\bin folder.

The options parameter was set to: JAVA_OPTS=-Xmx300m

After it we restarted the connector using the command: nctrl restart jdbc_connector_name

Immediately the documents started been processed with no error messages on the JDBC connector log.

See you,

Amadeu.