Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://softdevblogs.com/' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

Android changes for NDK developers

Android Developers Blog - Mon, 06/27/2016 - 23:53

Posted by Dmitry Malykhanov, Developer Advocate

Related to other improvements to the Android platform, the dynamic linker in Android M and N has stricter requirements for writing clean, cross-platform compatible native code in order to load. It is necessary that an application’s native code follows the rules and recommendations in order to ensure a smooth transition to recent Android releases.

Below we outline in detail each individual change related to native code loading, the consequences and steps you can take to avoid issues.

Required tools: there is an <arch>-linux-android-readelf binary (e.g. arm-linux-androideabi-readelf or i686-linux-android-readelf) for each architecture in the NDK (under toolchains/), but you can use readelf for any architecture, as we will be doing basic inspection only. On Linux you need to have the “binutils” package installed for readelf, and “pax-utils” for scanelf.

Private API (Enforced since API 24)

Native libraries must use only public API, and must not link against non-NDK platform libraries. Starting with API 24 this rule is enforced and applications are no longer able to load non-NDK platform libraries. The rule is enforced by the dynamic linker, so non-public libraries are not accessible regardless of the way code tries to load them: System.loadLibrary(...), DT_NEEDED entries, and direct calls to dlopen(...) will fail in exactly the same way.

Users should have a consistent app experience across updates, and developers shouldn’t have to make emergency app updates to handle platform changes. For that reason, we recommend against using private C/C++ symbols. Private symbols aren’t tested as part of the Compatibility Test Suite (CTS) that all Android devices must pass. They may not exist, or they may behave differently. This makes apps that use them more likely to fail on specific devices, or on future releases --- as many developers found when Android 6.0 Marshmallow switched from OpenSSL to BoringSSL.

In order to reduce the user impact of this transition, we’ve identified a set of libraries that see significant use from Google Play’s most-installed apps, and that are feasible for us to support in the short term (including libandroid_runtime.so, libcutils.so, libcrypto.so, and libssl.so). In order to give you more time to transition, we will temporarily support these libraries; so if you see a warning that means your code will not work in a future release -- please fix it now!

$ readelf --dynamic libBroken.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [libnativehelper.so]
 0x00000001 (NEEDED)                     Shared library: [libutils.so]
 0x00000001 (NEEDED)                     Shared library: [libstagefright_foundation.so]
 0x00000001 (NEEDED)                     Shared library: [libmedia_jni.so]
 0x00000001 (NEEDED)                     Shared library: [liblog.so]
 0x00000001 (NEEDED)                     Shared library: [libdl.so]
 0x00000001 (NEEDED)                     Shared library: [libz.so]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so]
 0x00000001 (NEEDED)                     Shared library: [libm.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so]

Potential problems: starting from API 24 the dynamic linker will not load private libraries, preventing the application from loading.

Resolution: rewrite your native code to rely only on public API. As a short term workaround, platform libraries without complex dependencies (libcutils.so) can be copied to the project. As a long term solution the relevant code must be copied to the project tree. SSL/Media/JNI internal/binder APIs should not be accessed from the native code. When necessary, native code should call appropriate public Java API methods.

A complete list of public libraries is available within the NDK, under platforms/android-API/usr/lib.

Note: SSL/crypto is a special case, applications must NOT use platform libcrypto and libssl libraries directly, even on older platforms. All applications should use GMS Security Provider to ensure they are protected from known vulnerabilities.

Missing Section Headers (Enforced since API 24)

Each ELF file has additional information contained in the section headers. These headers must be present now, because the dynamic linker uses them for sanity checking. Some developers try to strip them in an attempt to obfuscate the binary and prevent reverse engineering. (This doesn’t really help because it is possible to reconstruct the stripped information using widely-available tools.)

$ readelf --header libBroken.so | grep 'section headers'
  Start of section headers:          0 (bytes into file)
  Size of section headers:           0 (bytes)
  Number of section headers:         0
$

Resolution: remove the extra steps from your build that strip section headers.

Text Relocations (Enforced since API 23)

Starting with API 23, shared objects must not contain text relocations. That is, the code must be loaded as is and must not be modified. Such an approach reduces load time and improves security.

The usual reason for text relocations is non-position independent hand-written assembler. This is not common. Use the scanelf tool as described in our documentation for further diagnostics:

$ scanelf -qT libTextRel.so
  libTextRel.so: (memory/data?) [0x15E0E2] in (optimized out: previous simd_broken_op1) [0x15E0E0]
  libTextRel.so: (memory/data?) [0x15E3B2] in (optimized out: previous simd_broken_op2) [0x15E3B0]
[skipped the rest]

If you have no scanelf tool available, it is possible to do a basic check with readelf instead, look for either a TEXTREL entry or the TEXTREL flag. Either alone is sufficient. (The value corresponding to the TEXTREL entry is irrelevant and typically 0 --- simply the presence of the TEXTREL entry declares that the .so contains text relocations). This example has both indicators present:

$ readelf --dynamic libTextRel.so | grep TEXTREL
 0x00000016 (TEXTREL)                    0x0
 0x0000001e (FLAGS)                      SYMBOLIC TEXTREL BIND_NOW
$

Note: it is technically possible to have a shared object with the TEXTREL entry/flag but without any actual text relocations. This doesn’t happen with the NDK, but if you’re generating ELF files yourself make sure you’re not generating ELF files that claim to have text relocations, because the Android dynamic linker trusts the entry/flag.

Potential problems: Relocations enforce code pages being writable, and wastefully increase the number of dirty pages in memory. The dynamic linker has issued warnings about text relocations since Android K (API 19), but on API 23 and above it refuses to load code with text relocations.

Resolution: rewrite assembler to be position independent to ensure no text relocations are necessary. Check the Gentoo documentation for cookbook recipes.

Invalid DT_NEEDED Entries (Enforced since API 23)

While library dependencies (DT_NEEDED entries in the ELF headers) can be absolute paths, that doesn’t make sense on Android because you have no control over where your library will be installed by the system. A DT_NEEDED entry should be the same as the needed library’s SONAME, leaving the business of finding the library at runtime to the dynamic linker.

Before API 23, Android’s dynamic linker ignored the full path, and used only the basename (the part after the last ‘/’) when looking up the required libraries. Since API 23 the runtime linker will honor the DT_NEEDED exactly and so it won’t be able to load the library if it is not present in that exact location on the device.

Even worse, some build systems have bugs that cause them to insert DT_NEEDED entries that point to a file on the build host, something that cannot be found on the device.

$ readelf --dynamic libSample.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [libm.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so]
 0x00000001 (NEEDED)                     Shared library: [libdl.so]
 0x00000001 (NEEDED)                     Shared library:
[C:\Users\build\Android\ci\jni\libBroken.so]
$

Potential problems: before API 23 the DT_NEEDED entry’s basename was used, but starting from API 23 the Android runtime will try to load the library using the path specified, and that path won’t exist on the device. There are broken third-party toolchains/build systems that use a path on a build host instead of the SONAME.

Resolution: make sure all required libraries are referenced by SONAME only. It is better to let the runtime linker to find and load those libraries as the location may change from device to device.

Missing SONAME (Used since API 23)

Each ELF shared object (“native library”) must have a SONAME (Shared Object Name) attribute. The NDK toolchain adds this attribute by default, so its absence indicates either a misconfigured alternative toolchain or a misconfiguration in your build system. A missing SONAME may lead to runtime issues such as the wrong library being loaded: the filename is used instead when this attribute is missing.

$ readelf --dynamic libWithSoName.so | grep SONAME
 0x0000000e (SONAME)                     Library soname: [libWithSoName.so]
$

Potential problems: namespace conflicts may lead to the wrong library being loaded at runtime, which leads to crashes when required symbols are not found, or you try to use an ABI-incompatible library that isn’t the library you were expecting.

Resolution: the current NDK generates the correct SONAME by default. Ensure you’re using the current NDK and that you haven’t configured your build system to generate incorrect SONAME entries (using the -soname linker option).

Please remember, clean, cross-platform code built with a current NDK should have no issues on Android N. We encourage you to revise your native code build so that it produces correct binaries.

Categories: Programming

Android Mobile Vision restores operation and adds Text API

Android Developers Blog - Mon, 06/27/2016 - 21:07

Posted by Michael Sipe, Product Manager

As an important framework for finding objects in photos and video, Mobile Vision operation for Android devices is restored in Google Play Services v9.2.

This new version of Google Play Services fixes a download issue in Google Play Services v.9.0 that caused a service outage. See release notes for details.

We’re also pleased to announce the Text API, a new component for Android Mobile Vision.

The Text API’s optical character recognition technology reads Latin character text (e.g. English, Spanish, German, French, etc.) in photos and returns the text as well as the organizational structure (paragraphs, lines, words). Mobile apps can now:

  • Organize photos that contain text
  • Automate tedious data entry for credit cards, receipts, and business cards
  • Translate documents (along with the Cloud Translate API)
  • Keep track of real objects, such as reading the numbers on subway trains
  • Provide accessibility features

If you want to get started quickly, you can try our codelab which will get Android developers reading text with their apps in under an hour.

Like the Mobile Vision Face and Barcode components, the Text API runs on-device and is suitable for real-time applications. For more information, check out the Mobile Vision Developer site.

Categories: Programming

TensorFlow v0.9 now available with improved mobile support

Google Code Blog - Mon, 06/27/2016 - 19:56

Posted by Pete Warden, Software Engineer

When we started building TensorFlow, supporting mobile devices was a top priority. We were already supporting many of Google’s mobile apps like Translate, Maps, and the Google app, which use neural networks running on devices. We knew that we had to make mobile a first-class part of open source TensorFlow.

TensorFlow has been available to developers on Android since launch, and today we're happy to add iOS in v0.9 of TensorFlow, along with Raspberry Pi support and new compilation options.

To build TensorFlow on iOS we’ve created a set of scripts, including a makefile, to drive the cross-compilation process. The makefile can also help you build TensorFlow without using Bazel, which is not always available.

All this is in the latest TensorFlowdistribution. You can read more by visiting our Mobile TensorFlow guide and the documentation in our iOS samples and Android sample. The mobile samples allow you to classify images using the ImageNet Inception v1 classifier.

These mobile samples are just the beginning---we'd love your help and your contributions. Tag social media posts with #tensorflow so we can hear about your projects!

See the full TensorFlow 0.9.0 release notes here.

Categories: Programming

New Google Cast SDK released for Android and iOS

Google Code Blog - Mon, 06/27/2016 - 19:28

Posted by Adam Champy, Product Manager for Google Cast SDK

Google Cast makes it easy for developers to extend their mobile experience to the most beautiful screens and speakers in the home.

At Google I/O, we announced our new Google Cast SDK. This new SDK focuses on making development for Cast quicker, more reliable, and easier to maintain. We’ve introduced full state management that helps you implement the right abstraction between your app and Google Cast. We’ve also delivered a full Cast user experience, matching the Google Cast design checklist.

Today we are releasing this SDK for Android and iOS Senders, including an introductory video, full documentation, and reference sample apps and codelab tutorials for both platforms. Initial developer feedback is that first-time implementations can save significant development time compared with our previous SDKs.


A few things we’ve announced will be coming in the next few months, including a customizable Expanded Controller and adding customization to the Mini Controller, to help accelerate development even further.

Drop by our Cast developer site to learn about the new SDK and APIs, and join our developer community on Google+ at g.co/googlecastdev to discuss this with other developers.

Categories: Programming

Create Intelligent, Context-Aware Apps with the Google Awareness APIs

Android Developers Blog - Mon, 06/27/2016 - 18:29

Posted by Bhavik Singh, Product Manager

Last month at Google I/O 2016 we announced the new Google Awareness APIs, enabling your apps to intelligently react to user context using snapshots and fences with minimal impact on system resources.

Today we’re proud to announce that the Google Awareness API is available to all developers through Google Play services.

Using 7 different types of context—including location, weather, user activity, and nearby beacons—your app can better understand your users’ current situations, and use this information to provide optimized and customized experiences.

The Awareness API offers two ways to take advantage of context signals within your app:

  • The Snapshot API lets your app easily request information about the user's current context. For example, "give me the user's current location and the current weather conditions".
  • The Fence API lets your app react to changes in user’s context - and when it matches a certain set of conditions. For example, "tell me whenever the user is walking and their headphones are plugged in". Similar to the Geofencing API, once an awareness fence is registered, it can send callbacks to your app even when it's not running.

As a single, simplified surface, the Awareness APIs combine optimally processed context signals in new ways that were not previously possible, providing more accurate and insightful context cues, while also managing system resources to save battery and minimize bandwidth.

We’ve worked closely with some of our partners, who have already found amazing ways to integrate context awareness into their apps:

Trulia, an online residential real estate site, uses our Fence API to suggest open houses. When the weather is perfect and the user is walking around near a house they are interested in, Trulia sends a notification reminding them to stop by. This sort of tailored notification can help users engage with open houses at the perfect time for them.

SuperPlayer Music, on the other hand, uses our Snapshot API and Fence API to suggest the perfect music to match your mood. Whether you’re just finishing up a run and beginning to stretch, setting off on a long car ride, or just getting to the gym, their assistant can understand your context and suggest the right playlist for you.

With our initial set of signals and our awesome partners, we’re just getting started with the Awareness APIs. Join us on a journey to build tailored experiences within your apps, by getting started with the Google Awareness API developer documentation, and learn more by watching our Google I/O session


Categories: Programming

Reimagine Your Productivity at Getting Results.com

image

Imagine if you could master your motivation, your productivity, and your time management?

It’s time to get your game on.

I’ve completely revamped Getting Results.com (http://GettingResults.com) to help you think, feel, and do your best in any situation.

At Getting Results.com you can learn all about Agile Results.  It’s more than just a time management system or a productivity system.

It’s a personal results system for work and life.

Agile Results is a simple system for meaningful results.

It helps you respond to change while making things happen.  It helps you use your best energy for your best results, while you keep learning and improving.

What Will You Find at Getting Results.com?

Getting Results.com is a source of insight, inspiration, and action for mastering productivity, motivation, time management and more.

Here are some of the key things you’ll have access to:

  1. Get Started with Agile Results.
  2. Articles on Agile Results, Goals, Motivation, and More.
  3. 7 Day Agile Results Jumpstart, helps you take Agile Results for a test-drive and potentially have one of your best weeks.  Ever.
  4. 30 Days of Getting Results, this is advanced training that will help you get better results for work and life.
  5. Resources that include Cheat Sheets, Checklists, How Tos, Visuals, and more.

You can also read Success Stories to hear about how others have implemented and are using Agile Results.

Getting Started with Agile Results

While there are a lot of resources at Getting Results.com, the most important thing you can actually do is just get started with Agile Results:

Get Started with Agile Results

Take it for a test drive and see if you can create better results at work and in your life, the Agile Way.

Categories: Architecture, Programming

How Facebook Live Streams to 800,000 Simultaneous Viewers

Fewer companies know how to build world spanning distributed services than there are countries with nuclear weapons. Facebook is one of those companies and Facebook Live, Facebook’s new live video streaming product, is one one of those services.

Facebook CEO Mark Zuckerberg

The big decision we made was to shift a lot of our video efforts to focus on Live, because it is this emerging new format; not the kind of videos that have been online for the past five or ten years...We’re entering this new golden age of video. I wouldn’t be surprised if you fast-forward five years and most of the content that people see on Facebook and are sharing on a day-to-day basis is video.

If you are in the advertising business what could better than a supply of advertising ready content that is never ending, always expanding, and freely generated? It’s the same economics Google exploited when it started slapping ads on an exponentially growing web.

An example of Facebook’s streaming prowess is a 45 minute video of two people exploding a watermelon with rubber bands. It reached a peak of over 800,000 simultaneous viewers who also racked up over 300,000 comments. That’s the kind of viral scale you can generate with a social network of 1.5 billion users.

As a comparison The 2015 Super Bowl was watched by 114 million viewers with an average 2.36 million on the live stream. On Twitch there was a peak of 840,000 viewers at E3 2015. The September 16th Republican debate peaked at 921,000 simultaneous live streams.

So Facebook is right up there with the state of the art. Keep in mind Facebook would have a large number of other streams going on at the same time as well.

A Wired article quotes Chris Cox, Facebook’s chief product officer, who said Facebook:

  • Has more than a hundred people working on Live. (it started with ~12 and now there are more than 150 engineers on the project)

  • Needs to be able to serve up millions of simultaneous streams without crashing.

  • Need to be able to support millions of simultaneous viewers on a stream, as well as seamless streams across different devices and service providers around the world.

Cox said that “It turns out it’s a really hard infrastructure problem.”

Wouldn't it be interesting if we had some details about how that infrastructure problem was solved? Woe is we. But wait, we do!

 Federico Larumbe from Facebook’s Traffic Team, which works on the caching software powering Facebook’s CDN and the Global Load Balancing system, gave an excellent talk: Scaling Facebook Live, where he shares some details about how Live works.

Here’s my gloss on the talk. It’s impressive.

Origin Story
Categories: Architecture

Project Bloks: Making code physical for kids

Google Code Blog - Mon, 06/27/2016 - 16:46

Originally posted on Google Research Blog


Posted by Steve Vranakis and Jayme Goldstein, Executive Creative Director and Project Lead, Google Creative Lab
At Google, we’re passionate about empowering children to create and explore with technology. We believe that when children learn to code, they’re not just learning how to program a computer—they’re learning a new language for creative expression and are developing computational thinking: a skillset for solving problems of all kinds.
In fact, it’s a skillset whose importance is being recognised around the world—from President Obama’s CS4All program to the inclusion of Computer Science in the UK National Curriculum. We’ve long supported and advocated the furthering of CS education through programs and platforms such as Blockly, Scratch Blocks, CS First and Made w/ Code.
Today, we’re happy to announce Project Bloks, a research collaboration between Google, Paulo Blikstein (Stanford University) and IDEO with the goal of creating an open hardware platform that researchers, developers and designers can use to build physical coding experiences. As a first step, we’ve created a system for tangible programming and built a working prototype with it. We’re sharing our progress before conducting more research over the summer to inform what comes next.
Physical codingKids are inherently playful and social. They naturally play and learn by using their hands, building stuff and doing things together. Making code physical - known as tangible programming - offers a unique way to combine the way children innately play and learn with computational thinking.
Project Bloks is preceded and shaped by a long history of educational theory and research in the area of hands-on learning. From Friedrich Froebel, Maria Montessori and Jean Piaget’s pioneering work in the area of learning by experience, exploration and manipulation, to the research started in the 1970s by Seymour Papert and Radia Perlman with LOGO and TORTIS. This exploration has continued to grow and includes a wide range of research and platforms.
However, designing kits for tangible programming is challenging—requiring the resources and time to develop both the software and the hardware. Our goal is to remove those barriers. By creating an open platform, Project Bloks will allow designers, developers and researchers to focus on innovating, experimenting and creating new ways to help kids develop computational thinking. Our vision is that, one day, the Project Bloks platform becomes for tangible programming what Blockly is for on-screen programming. The Project Bloks systemWe’ve designed a system that developers can customise, reconfigure and rearrange to create all kinds of different tangible programming experiences. A birdseye view of the customisable and reconfigurable Project Bloks systemThe Project Bloks system is made up of three core components the “Brain Board”, “Base Boards” and “Pucks”. When connected together they create a set of instructions which can be sent to connected devices, things like toys or tablets, over wifi or Bluetooth. The three core components of the Project Bloks systemPucks: abundant, inexpensive, customisable physical instructionsPucks are what make the Project Bloks system so versatile. They help bring the infinite flexibility of software programming commands to tangible programming experiences. Pucks can be programmed with different instructions, such as ‘turn on or off’, ‘move left’ or ‘jump’. They can also take the shape of many different interactive forms—like switches, dials or buttons. With no active electronic components, they’re also incredibly cheap and easy to make. At a minimum, all you'd need to make a puck is a piece of paper and some conductive ink. Pucks allow for the creation and customisation of endless amount of different domain-specific physical instructions cheaply and easily.Base Boards: a modular design for diverse tangible programming experiencesBase Boards read a Puck’s instruction through a capacitive sensor. They act as a conduit for a Puck’s command to the Brain Board. Base Boards are modular and can be connected in sequence and in different orientations to create different programming flows and experiences. The modularity of the Base Boards means they can be arranged in different configurations and flowsEach Base Board is fitted with a haptic motor and LEDs that can be used to give end-users real time feedback on their programming experience. The Base Boards can also trigger audio feedback from the Brain Board’s built-in speaker. Brain Board: control any device that has an API over WiFi or BluetoothThe Brain Board is the processing unit of the system, built on a Raspberry Pi Zero. It also provides the other boards with power, and contains an API to receive and send data to the Base Boards. It sends the Base Boards’ instructions to any device with WiFi or Bluetooth connectivity and an API. As a whole, the Project Bloks system can take on different form factors and be made out of different materials. This means developers have the flexibility to create diverse experiences that can help kids develop computational thinking: from composing music using functions to playing around with sensors or anything else they care to invent. The Project Bloks system can be used to create all sorts of different physical programming experiences for kidsThe Coding KitTo show how designers, developers, and researchers might make use of system, the Project Bloks team worked with IDEO to create a reference device, called the Coding Kit. It lets kids learn basic concepts of programming by allowing them to put code bricks together to create a set of instructions that can be sent to control connected toys and devices—anything from a tablet, to a drawing robot or educational tools for exploring science like LEGO® Education WeDo 2.0. What’s next?We are looking for participants (educators, developers, parents and researchers) from around the world who would like to help shape the future of Computer Science education by remotely taking part in our research studies later in the year. If you would like to be part of our research study or simply receive updates on the project, please sign up. If you want more context and detail on Project Bloks, you can read our position paper. Finally, a big thank you to the team beyond Google who’ve helped us get this far—including the pioneers of tangible learning and programming who’ve inspired us and informed so much of our thinking.
Categories: Programming

The Bestselling Software Intended for People Who Couldn’t Use It.

James Bach’s Blog - Mon, 06/27/2016 - 16:26

In 1983, my boss, Dale Disharoon, designed a little game called Alphabet Zoo. My job was to write the Commodore 64 and Apple II versions of that game. Alphabet Zoo is a game for kids who are learning to read. The child uses a joystick to move a little character through a maze, collecting letters to spell words.

We did no user testing on this game until the very day we sent the final software to the publisher. On that day, we discovered that out target users (five year-olds) did not have the ability to use a joystick well enough to play the game. They just got frustrated and gave up.

We shipped anyway.

This game became a bestseller.

alphabet-zoo

It was placed on a list of educational games recommended by the National Education Association.

alphabet-zoo-list

Source: Information Please Almanac, 1986

So, how to explain this?

Some years later, when I became a father, I understood. My son was able to play a lot of games that were too hard for him because I operated the controls. I spoke to at least one dad who did exactly that with Alphabet Zoo.

I guess the moral of the story is: we don’t necessarily know the value of our own creations.

Categories: Testing & QA

SPaMCAST 400 – Jim Benson, Personal Kanban and More

SPaMCAST Logo

http://www.spamcast.net

Listen Now

Subscribe on iTunes                   Check out the podcast on Google Play Music

Software Process and Measurement Cast 400 features our interview with Jim Benson. Jim and I talked about personal Kanban, micromanagement, work-in-process limits, pattern matching, pomodoro and more. A great interview to cap our first 400 episodes!

Jim’s career path has taken him through government agencies, Fortune 10 corporations, and start-ups. Through them all his passion has remained consistent – applying new technologies to work groups. In each case asking how they can be leveraged to collaborate and cooperate more effectively. Jim loves ideas, creation, and building opportunities. He loves working with teams who are passionate about the future, pushing boundaries, and inclusion. His goal with all technologies is to increase beneficial contact between people and reduce the bureaucratic noise which so often tends to increase costs and destroy creativity.

Jim is the author of the Shingo Research Award winning book Personal Kanban (use the link to support the podcast) . He is a noted expert in business process, personal work management, and the application of Lean to personal work and life. Jim believes that the best process is the least process necessary to achieve goals. He has zero tolerance for process waste.

All said, Jim enjoys helping people and teams work out sticky problems, an advocate of people actually seeing their work, and inventing new ways to work at the intersection of Lean thinking, brain science, and leadership.

Contact Jim

Twitter: https://twitter.com/ourfounder

LinkedIn: https://www.linkedin.com/in/jimbenson

Personal Kanban: http://www.personalkanban.com/pk/#sthash.MtOA96sV.dpbs

Modus Cooperandi: http://moduscooperandi.com/

Re-Read Saturday News

This week we continue the Re-read Saturday of  Kent Beck’s XP Explained, Second Edition with a discussion of Chapters 2 and 3.  The first two chapters in section One provide us with an overview of the conceptual framework that underpins XP.  Use the link to XP Explained in the show notes when you buy your copy to read along to support both the blog and podcast. Visit the Software Process and Measurement Blog (www.tcagley.wordpress.com) to catch up on past installments of Re-Read Saturday.

Next SPaMCAST

The next Software Process and Measurement Cast will feature our essay on listening.  Effective listening is CRITICAL to every aspect of software development  and maintenance. Listening is a complex set of tasks that is more than simply receiving audio data. You also need to interpret that data.

We will also have columns from Jeremy Berriault who brings us the QA Corner and Jon M. Quigley’s second column which covers the gamut of product development.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, for you or your team.” Support SPaMCAST by buying the book here. Available in English and Chinese.


Categories: Process Management

SPaMCAST 400 – Jim Benson, Personal Kanban and More

Software Process and Measurement Cast - Sun, 06/26/2016 - 22:00

Software Process and Measurement Cast 400 features our interview with Jim Benson. Jim and I talked about personal Kanban, micromanagement, work-in-process limits, pattern matching, pomodoro and more. A great interview to cap our first 400 episodes!

Jim’s career path has taken him through government agencies, Fortune 10 corporations, and start-ups. Through them all his passion has remained consistent – applying new technologies to work groups. In each case asking how they can be leveraged to collaborate and cooperate more effectively. Jim loves ideas, creation, and building opportunities. He loves working with teams who are passionate about the future, pushing boundaries, and inclusion. His goal with all technologies is to increase beneficial contact between people and reduce the bureaucratic noise which so often tends to increase costs and destroy creativity.

Jim is the author of the Shingo Research Award winning book Personal Kanban (use the link to buy a copy and support the podcast). He is a noted expert in business process, personal work management, and the application of Lean to personal work and life. Jim believes that the best process is the least process necessary to achieve goals. He has zero tolerance for process waste.

All said, Jim enjoys helping people and teams work out sticky problems, an advocate of people actually seeing their work, and inventing new ways to work at the intersection of Lean thinking, brain science, and leadership.

 

Contact Jim

Twitter: https://twitter.com/ourfounder

LinkedIn: https://www.linkedin.com/in/jimbenson

Personal Kanban: http://www.personalkanban.com/pk/#sthash.MtOA96sV.dpbs

Modus Cooperandi: http://moduscooperandi.com/

 

Re-Read Saturday News

This week we continue the Re-read Saturday of  Kent Beck’s XP Explained, Second Edition with a discussion of Chapters 2 and 3.  The first two chapters in section One provide us with an overview of the conceptual framework that underpins XP.  Use the link to XP Explained in the show notes when you buy your copy to read along to support both the blog and podcast. Visit the Software Process and Measurement Blog (www.tcagley.wordpress.com) to catch up on past installments of Re-Read Saturday.

 

Next SPaMCAST

The next Software Process and Measurement Cast will feature our essay on listening.  Effective listening is CRITICAL to every aspect of software development  and maintenance. Listening is a complex set of tasks that is more than simply receiving audio data. You also need to interpret that data.

We will also have columns from Jeremy Berriault who brings us the QA Corner and Jon M. Quigley’s second column which covers the gamut of product development.

 

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, for you or your team.” Support SPaMCAST by buying the book here. Available in English and Chinese.

Categories: Process Management

Decision Making in the Presence of Uncertainty

Herding Cats - Glen Alleman - Sun, 06/26/2016 - 19:08

There's another rash of Twitter posters supporting the conjecture that decisions can be made about how to spend other people's money without estimating the impact and outcome of that decision. This is a core premise of #NoEstimates from the Original Poster

No Estimates

Here's a starting point to learn that is simply not true...

There are 100's more books, papers, conference proceedings on the topic of decision making in the presence of uncertanty.

It comes down to a simple concept

All project work is uncertain. Reducible uncertainties (epistemic) and irreducible uncertainties (aleatory) are present on all projects. When money is being provided to develop software, those providing the money have a reasonable expectation to know something about when you will be providing the value in exchange for that money, how much money will be required to provide that value, and what capabilities will be provided in exchange for that money. The field of study where these questions and answers live is microeconomics of decision making. Software development decision making is a well studied subset of that field.

When it is conjectured decisions can be made in the presence of uncertanty without estimates - like the Original Poster did, and there is no supporting theory, principle, practices, or evidence of how that can be done - run away - it's a bogus claim. 

Related articles Making Decisions In The Presence of Uncertainty Essential Reading List for Managing Other People's Money Herding Cats: Decision Making in the Presence of Uncertainty Herding Cats: Thoughts on Suggestions That Violate Principles of Finance, Probability, and Statistics Making Conjectures Without Testable Outcomes
Categories: Project Management

Extreme Programming Explained: Embrace Change Second Edition, Week 2 (Chapters 2 -3)

XP Explained

This week we begin Section 1of Kent Beck’s Extreme Programing Explained, Second Edition (2005), titled Exploring XP, by tackling Chapters two and three.  The first two chapters in section One provide us with an overview of the conceptual framework that underpins XP. 

Chapter 2: Learning to Drive.  In this chapter Beck challenges us to consider that developing a product or writing code is like driving.  The metaphor of driving is used because driving is not a one-time event where you start the car and point it in the right direction then take a nap.  Rather, driving is a process of constantly paying attention, gathering feedback and making a series of small changes so that you arrive at your goal healthy and whole.  As noted in the preface, the core XP paradigm is “stay aware, adapt, change.” Tying the driving metaphor and XP together is the common problem of mushy requirements. In XP customers drive the content of the system and steer by getting feedback and adapting based on short iterations. 

As a veteran of the quality movement of the 1990’s, XP with its “stay aware, adapt, change” empirical model is easily recognizable to me as continuous improvement model that improves effectiveness, communication, competence, and productivity.

Chapter 3: Values, Principles, and Practices. XP, a methodology, represents a very different approach to software development. In order to effectively use the practices that make up XP requires adopting a set of values that are implemented based on a set of principles. Practices are the things we do, and therefore, are clear and objective. They are done or they are not done.  For example, a stand-up meeting is a practice; anyone can easily confirm whether a standup meeting is being held. In the book, Beck used the difference between a person who putters around the garden and a master gardener to illustrate the reasons why learning techniques and practices do not yield mastery. Values allow the practitioner to understand the environment, apply and evolve practices when the context demands. No framework stands as merely a set of rote practices even it might seem like just doing practices leads to a useful outcome. This why there are arguments about the ideas of “being Agile” rather than just “doing Agile.”  The lure of  just learning and applying practices without a framework of values and principles is not limited to software development. On my bookshelf there is a book titled, You Can’t Teach A Kid To Ride A Bike At A Seminar. The book is on sales (I believe everyone should have sales training).  Like most how-to books it is full of practices and techniques, with one big difference. The book makes the point that without a set of values and principles we are just robots that don’t have the level of knowledge and understanding needed to apply those practices when the context changes.  Values, principles, and principles are necessary to be a gardener, salesperson, or developer.

Values are an overarching set of rules define the boundaries of behavior. Values define what we believe in and are used to judge what we see, think, and do.  Most cognitive biases are driven by the values a person or team holds. A team that values being a process more than people will act much differently than a team with the opposite set of values.  Defining explicit values provide humans with purpose and direction.

Principles act as the connecting tissue between practices and values.  Principles provide the context-specific guidelines for implementing practices based on the values the team holds.  Beck uses the metaphor of a bridge between values and practices. 

I often talk to people that begin a transformation by adopting practices before thinking about values and principles. Perhaps a small group of true believers have drunk the Koolaid and have institutionalized the values and principles that are at the core of the framework.  However, that belief structure does not extend very away from that core group.  This approach yields highly variable results and rarely delivers long-term change. This variability occurs whether discussing Agile, the CMMI or a new sales approach. The rationale often seems to be that by adopting practices it is clear that “change” has occurred because it easy to see actions.  However because practices are situational, without the guidance of internalized values and principles it is difficult for practices to mold to situations.  When practice doesn’t seem to fit, domain specific guidelines are needed or they will be abandoned. 

Previous installment of Extreme Programing Explained, Second Edition (2005) on Re-read Saturday:

Extreme Programming Explained: Embrace Change Second Edition Week 1, Preface and Chapter 1


Categories: Process Management

Improving Stability with Private C/C++ Symbol Restrictions in Android N

Android Developers Blog - Sat, 06/25/2016 - 02:04

Posted by Dimitry Ivanov & Elliott Hughes, Software Engineers

As documented in the Android N behavioral changes, to protect Android users and apps from unforeseen crashes, Android N will restrict which libraries your C/C++ code can link against at runtime. As a result, if your app uses any private symbols from platform libraries, you will need to update it to either use the public NDK APIs or to include its own copy of those libraries. Some libraries are public: the NDK exposes libandroid, libc, libcamera2ndk, libdl, libGLES, libjnigraphics, liblog, libm, libmediandk, libOpenMAXAL, libOpenSLES, libstdc++, libvulkan, and libz as part of the NDK API. Other libraries are private, and Android N only allows access to them for platform HALs, system daemons, and the like. If you aren’t sure whether your app uses private libraries, you can immediately check it for warnings on the N Developer Preview.

We’re making this change because it’s painful for users when their apps stop working after a platform update. Whether they blame the app developer or the platform, everybody loses. Users should have a consistent app experience across updates, and developers shouldn’t have to make emergency app updates to handle platform changes. For that reason, we recommend against using private C/C++ symbols. Private symbols aren’t tested as part of the Compatibility Test Suite (CTS) that all Android devices must pass. They may not exist, or they may behave differently. This makes apps that use them more likely to fail on specific devices, or on future releases — as many developers found when Android 6.0 Marshmallow switched from OpenSSL to BoringSSL.

You may be surprised that there’s no STL in the list of NDK libraries. The three STL implementations included in the NDK — the LLVM libc++, the GNU STL, and libstlport — are intended to be bundled with your app, either by statically linking into your library, or by inclusion as a separate shared library. In the past, some developers have assumed that they didn’t need to package the library because the OS itself had a copy. This assumption is incorrect: a particular STL implementation may disappear (as was the case with stlport, which was removed in Marshmallow), may never have been available (as is the case with the GNU STL), or it may change in ABI incompatible ways (as is the case with the LLVM libc++).

In order to reduce the user impact of this transition, we’ve identified a set of libraries that see significant use from Google Play’s most-installed apps, and that are feasible for us to support in the short term (including libandroid_runtime.so, libcutils.so, libcrypto.so, and libssl.so). For legacy code in N, we will temporarily support these libraries in order to give you more time to transition. Note that we don't intend to continue this support in any future Android platform release, so if you see a warning that means your code will not work in a future release — please fix it now!

Table 1. What to expect if your app is linking against private native libraries.

Libraries App's targetSdkVersion Runtime access via dynamic linker Impact, N Developer Preview Impact, Final N Release Impact, future platform version NDK Public Any Accessible Private (graylist) <=23 Temporarily accessible Warning / Toast Warning Error >=24 Restricted Error Error Error Private (all other)> Any Restricted Error Error Error What behavior will I see?

Please test your app during the N Previews.

N Preview behavior
  • All public NDK libraries (libandroid, libc, libcamera2ndk, libdl, libGLES, libjnigraphics, liblog, libm, libmediandk, libOpenMAXAL, libOpenSLES, libstdc++, libvulkan, and libz), plus libraries that are part of your app are accessible.
  • For all other libraries you’ll see a warning in logcat and a toast on the display. This will happen only if your app’s targetSdkVersion is less than N. If you change your manifest to target N, loading will fail: Java’s System.loadLibrary will throw, and C/C++’s dlopen(3) will return NULL.

Test your apps on the Developer Preview — if you see a toast like this one, your app is accessing private native APIs. Please fix your code soon!

N Final Release behavior
  • All NDK libraries (libandroid, libc, libcamera2ndk, libdl, libGLES, libjnigraphics, liblog, libm, libmediandk, libOpenMAXAL, libOpenSLES, libstdc++, libvulkan, and libz), plus libraries that are part of your app are accessible.
  • For the temporarily accessible libraries (such as libandroid_runtime.so, libcutils.so, libcrypto.so, and libssl.so), you’ll see a warning in logcat for all API levels before N, but loading will fail if you update your app so that its targetSdkVersion is N or later.
  • Attempts to load any other libraries will fail in the final release of Android N, even if your app is targeting a pre-N platform version.
Future platform behavior
  • In O, all access to the temporarily accessible libraries will be removed. As a result, you should plan to update your app regardless of your targetSdkVersion prior to O. If you believe there is missing functionality from the NDK API that will make it impossible for you to transition off a temporarily accessible library, please file a bug here.
What do the errors look like?

Here’s some example logcat output from an app that hasn’t bumped its target SDK version (and so the restriction isn’t fully enforced because this is only the developer preview):

03-21 17:07:51.502 31234 31234 W linker  : library "libandroid_runtime.so"
("/system/lib/libandroid_runtime.so") needed or dlopened by
"/data/app/com.popular-app.android-2/lib/arm/libapplib.so" is not accessible
for the namespace "classloader-namespace" - the access is temporarily granted
as a workaround for http://b/26394120

This is telling you that your library “libapplib.so” refers to the library “libandroid_runtime.so”, which is a private library.

When Android N ships, or if you set your target SDK version to N now, you’ll see something like this if you try to use System.loadLibrary from Java:

java.lang.UnsatisfiedLinkError: dlopen failed: library "libcutils.so"
("/system/lib/libcutils.so") needed or dlopened by "/system/lib/libnativeloader.so"
is not accessible for the namespace "classloader-namespace"
  at java.lang.Runtime.loadLibrary0(Runtime.java:977)
  at java.lang.System.loadLibrary(System.java:1602)

If you’re using dlopen(3) from C/C++ you’ll get a NULL return and dlerror(3) will return the same “dlopen failed...” string as shown above.

For more information about how to check if your app is using private symbols, see the FAQ on developer.android.com.

Categories: Programming

#NoEstimates Book Review - Part 1

Herding Cats - Glen Alleman - Sat, 06/25/2016 - 01:17

I've started reading Vasco's book #NoEstimates and will write a detailed deconstruction. I got the Kindle version, so I have a $10 investment at risk. Let's start with some graphs that have been around and their misinformation that forms the basis of the book.

The Chaos Report graph,is the 1st one. This graph is from old 2004 numbers. That's 12 year old numbers. Many times the books uses 12, 16, even 25 year old reports as the basis of the suggestion that Not Estimating fixes the problems in the reports. The Chaos reports have been thoroughly debunked as self selected samples using uncalibrated surveys for the units of measure for project failure. Here's a few comments on the Standish reporting process. But first remember, Standish does not say what the units of measure are for Success, Challenged, or Failure. Without the units of measure, the actual statistics of the projects and the statistical ranges of those projects for each of the three categories, the units as essentially bogus. Good fodder for selling consulting services or for use by those with an idea to sell, but worthless for decision making about the root cause of Failure, Challenged, or even Success. Any undergraduate design of experiments class would have all that information in the public. 

So the 1st thing to read when you encounter data like this is Project Success: A Multidimensional Strategic Concept, Aaron J. Shenhar, Dov Dvir, Ofer Levy and Alan C. Maltz. Only then start to assess the numbers. Most likely, like the numbers in the book, they're not credible to support the hypothesis. Which by the way there is no hypothesis for you can make decisions in the presence of uncertanty without estimating

So let's look further at the difficulties with Standish and why NOT to use it as the basis of a conjecture

A simple google search would have found all this research and many many more. I get the sense V didn't do his homework. The bibliography has very few references to actually estimating, no actual estimating books, papers, or research sites. Just personal anecdotes from a  set of experiences as a developer.

The Standish Report failure mode is described in Darrell Huff's How to Lie With Statisticsself select the samples from the survey. Standish does not provide any population statistics for their survey.

  • How many surveys were set out?
  • How many responded?
  • Of those responding, how many were statistically significant?
  • What does it means in terms of actual measures of performance to the troubled?
  • If the project was over budget, was the needed budget estimate correct?
  • If the project was late, was the original schedule credible?

None of these questions are answered in Standish reports. No Estimate picks these serious statistical sampling error up and uses the, as the basis of the pure-conjecture that Not Estimating is going to fix the problems. This would garner a high school D is the Stats class.

Screen Shot 2016-06-24 at 2.08.13 PM

Next comes a chart that makes a similar error. This reference is from Steve McConnell's book, but is actually from another source. The No Estimates book does a poor job of keeping the references straight. It is common to misattribute a report, a graph, even a phrase. The book needs a professional editor.

The graph below is used to show that estimates are usually wrong. But there is a critical misunderstanding about the data.

  • It starts with a straight line called perfect accuracy/. First there are two attributes of any estimate. Accuracy and precision. There is no such thing as perfect accuracy in the estimating business. An estimate is an approximation of reality.
  • The sample projects show they did not meet the perfect accuracy - whatever that might have been. This knowledge can only be obtained after the work has been done. Either at the end or during the project - cumulative to date
  • But there are two sources of the variances of estimates and actuals

Screen Shot 2016-06-24 at 2.08.06 PM

I'm in the early parts of the book and already have a half dozen pages of notes for either fallacies, incorrect principles, 30 year old references, and other serious mistakes on understanding on how decisions are made in the presence of uncertainty. My short assessment is 

It's a concept built on sand.

Related articles Myth's Abound All Project Numbers are Random Numbers - Act Accordingly The Failure of Open Loop Thinking How to "Lie" with Statistics Statistical Significance How to Estimate Software Development Capabilities Based Planning IT Risk Management Why Guessing is not Estimating and Estimating is not Guessing Making Conjectures Without Testable Outcomes
Categories: Project Management

Grow your business on Google Play with help from the new Playbook for Developers app

Android Developers Blog - Fri, 06/24/2016 - 18:08

Posted by Dom Elliott, the Google Play team

Today, the Playbook for Developers mobile app is now generally available for Android devices. The app helps you stay up-to-date with the features and best practices to grow your business on Google Play. Thanks to all our beta testers over the last six weeks whose feedback helped us tweak and refine the app in preparation for launch.

Here’s how you read and watch content in the Playbook for Developers app:

  • Choose topics relating to your business interests to personalize My Playbook with curated articles and videos from Google and experts across the web.
  • Explore the in-depth guide to Google’s developer products, with articles grouped by what you’re trying to do: develop, launch, engage, grow, and earn.
  • Take actions on items – complete, share, save, or dismiss them – and read your Saved articles later, including offline if they’re written in the app. A data connection will be needed to read articles and videos from across the web.

The app supports Android 5.0 and above. If you're on an older device, check out our ebook, The Secrets to App Success on Google Play. We will be adding and updating content in the app to help you stay up-to-date and grow your business. Get the Playbook for Developers app today and then give us your feedback. The app is also available in the following languages: Bahasa Indonesia, Deutsch, español (Latinoamérica), le français, português do Brasil, tiếng Việt, русский язы́к, 한국어, 中文 (简体), 中文 (繁體), and 日本語.

This is the second app we’ve released for Google Play developers. Get the Google Play Developer Console app to review your app's performance statistics and financial data, get notified about your app's status and publishing changes, and read and reply to user reviews on the go.

Categories: Programming

Unix: Find files greater than date

Mark Needham - Fri, 06/24/2016 - 17:56

For the latter part of the week I’ve been running some tests against Neo4j which generate a bunch of log files and I wanted to filter those files based on the time they were created to do some further analysis.

This is an example of what the directory listing looks like:

$ ls -alh foo/database-agent-*
-rw-r--r--  1 markneedham  wheel   2.5K 23 Jun 14:00 foo/database-agent-mac17f73-1-logs-archive-201606231300176.tar.gz
-rw-r--r--  1 markneedham  wheel   8.6K 23 Jun 11:49 foo/database-agent-mac19b6b-1-logs-archive-201606231049507.tar.gz
-rw-r--r--  1 markneedham  wheel   8.6K 23 Jun 11:49 foo/database-agent-mac1f427-1-logs-archive-201606231049507.tar.gz
-rw-r--r--  1 markneedham  wheel   2.5K 23 Jun 14:00 foo/database-agent-mac29389-1-logs-archive-201606231300176.tar.gz
-rw-r--r--  1 markneedham  wheel    11K 23 Jun 13:44 foo/database-agent-mac3533f-1-logs-archive-201606231244152.tar.gz
-rw-r--r--  1 markneedham  wheel   4.8K 23 Jun 14:00 foo/database-agent-mac35563-1-logs-archive-201606231300176.tar.gz
-rw-r--r--  1 markneedham  wheel   3.8K 23 Jun 13:44 foo/database-agent-mac35f7e-1-logs-archive-201606231244165.tar.gz
-rw-r--r--  1 markneedham  wheel   4.8K 23 Jun 14:00 foo/database-agent-mac40798-1-logs-archive-201606231300176.tar.gz
-rw-r--r--  1 markneedham  wheel    12K 23 Jun 13:44 foo/database-agent-mac490bf-1-logs-archive-201606231244151.tar.gz
-rw-r--r--  1 markneedham  wheel   2.5K 23 Jun 14:00 foo/database-agent-mac5f094-1-logs-archive-201606231300189.tar.gz
-rw-r--r--  1 markneedham  wheel   5.8K 23 Jun 14:00 foo/database-agent-mac636b8-1-logs-archive-201606231300176.tar.gz
-rw-r--r--  1 markneedham  wheel   9.5K 23 Jun 11:49 foo/database-agent-mac7e165-1-logs-archive-201606231049507.tar.gz
-rw-r--r--  1 markneedham  wheel   2.7K 23 Jun 11:49 foo/database-agent-macab7f1-1-logs-archive-201606231049507.tar.gz
-rw-r--r--  1 markneedham  wheel   2.8K 23 Jun 13:44 foo/database-agent-macbb8e1-1-logs-archive-201606231244151.tar.gz
-rw-r--r--  1 markneedham  wheel   3.1K 23 Jun 11:49 foo/database-agent-macbcbe8-1-logs-archive-201606231049520.tar.gz
-rw-r--r--  1 markneedham  wheel    13K 23 Jun 13:44 foo/database-agent-macc8177-1-logs-archive-201606231244152.tar.gz
-rw-r--r--  1 markneedham  wheel   3.8K 23 Jun 13:44 foo/database-agent-maccd92c-1-logs-archive-201606231244151.tar.gz
-rw-r--r--  1 markneedham  wheel   3.9K 23 Jun 13:44 foo/database-agent-macdf24f-1-logs-archive-201606231244165.tar.gz
-rw-r--r--  1 markneedham  wheel   3.1K 23 Jun 11:49 foo/database-agent-mace075e-1-logs-archive-201606231049520.tar.gz
-rw-r--r--  1 markneedham  wheel   3.1K 23 Jun 11:49 foo/database-agent-mace8859-1-logs-archive-201606231049507.tar.gz

I wanted to split the files in half so that I could have the ones created before and after 12pm on the 23rd June.

I discovered that this type of filtering is actually quite easy to do with the ‘find’ command. So if I want to get the files after 12pm I could write the following:

$ find foo -name database-agent* -newermt "Jun 23, 2016 12:00" -ls
121939705        8 -rw-r--r--    1 markneedham      wheel                2524 23 Jun 14:00 foo/database-agent-mac17f73-1-logs-archive-201606231300176.tar.gz
121939704        8 -rw-r--r--    1 markneedham      wheel                2511 23 Jun 14:00 foo/database-agent-mac29389-1-logs-archive-201606231300176.tar.gz
121934591       24 -rw-r--r--    1 markneedham      wheel               11294 23 Jun 13:44 foo/database-agent-mac3533f-1-logs-archive-201606231244152.tar.gz
121939707       16 -rw-r--r--    1 markneedham      wheel                4878 23 Jun 14:00 foo/database-agent-mac35563-1-logs-archive-201606231300176.tar.gz
121934612        8 -rw-r--r--    1 markneedham      wheel                3896 23 Jun 13:44 foo/database-agent-mac35f7e-1-logs-archive-201606231244165.tar.gz
121939708       16 -rw-r--r--    1 markneedham      wheel                4887 23 Jun 14:00 foo/database-agent-mac40798-1-logs-archive-201606231300176.tar.gz
121934589       24 -rw-r--r--    1 markneedham      wheel               12204 23 Jun 13:44 foo/database-agent-mac490bf-1-logs-archive-201606231244151.tar.gz
121939720        8 -rw-r--r--    1 markneedham      wheel                2510 23 Jun 14:00 foo/database-agent-mac5f094-1-logs-archive-201606231300189.tar.gz
121939706       16 -rw-r--r--    1 markneedham      wheel                5912 23 Jun 14:00 foo/database-agent-mac636b8-1-logs-archive-201606231300176.tar.gz
121934588        8 -rw-r--r--    1 markneedham      wheel                2895 23 Jun 13:44 foo/database-agent-macbb8e1-1-logs-archive-201606231244151.tar.gz
121934590       32 -rw-r--r--    1 markneedham      wheel               13427 23 Jun 13:44 foo/database-agent-macc8177-1-logs-archive-201606231244152.tar.gz
121934587        8 -rw-r--r--    1 markneedham      wheel                3882 23 Jun 13:44 foo/database-agent-maccd92c-1-logs-archive-201606231244151.tar.gz
121934611        8 -rw-r--r--    1 markneedham      wheel                3970 23 Jun 13:44 foo/database-agent-macdf24f-1-logs-archive-201606231244165.tar.gz

And to get the ones before 12pm:

$ find foo -name database-agent* -not -newermt "Jun 23, 2016 12:00" -ls
121879391       24 -rw-r--r--    1 markneedham      wheel                8856 23 Jun 11:49 foo/database-agent-mac19b6b-1-logs-archive-201606231049507.tar.gz
121879394       24 -rw-r--r--    1 markneedham      wheel                8772 23 Jun 11:49 foo/database-agent-mac1f427-1-logs-archive-201606231049507.tar.gz
121879390       24 -rw-r--r--    1 markneedham      wheel                9702 23 Jun 11:49 foo/database-agent-mac7e165-1-logs-archive-201606231049507.tar.gz
121879393        8 -rw-r--r--    1 markneedham      wheel                2812 23 Jun 11:49 foo/database-agent-macab7f1-1-logs-archive-201606231049507.tar.gz
121879413        8 -rw-r--r--    1 markneedham      wheel                3144 23 Jun 11:49 foo/database-agent-macbcbe8-1-logs-archive-201606231049520.tar.gz
121879414        8 -rw-r--r--    1 markneedham      wheel                3131 23 Jun 11:49 foo/database-agent-mace075e-1-logs-archive-201606231049520.tar.gz
121879392        8 -rw-r--r--    1 markneedham      wheel                3130 23 Jun 11:49 foo/database-agent-mace8859-1-logs-archive-201606231049507.tar.gz

Or we could even find the ones last modified between 12pm and 2pm:

$ find foo -name database-agent* -not -newermt "Jun 23, 2016 14:00" -newermt "Jun 23, 2016 12:00" -ls
121934591       24 -rw-r--r--    1 markneedham      wheel               11294 23 Jun 13:44 foo/database-agent-mac3533f-1-logs-archive-201606231244152.tar.gz
121934612        8 -rw-r--r--    1 markneedham      wheel                3896 23 Jun 13:44 foo/database-agent-mac35f7e-1-logs-archive-201606231244165.tar.gz
121934589       24 -rw-r--r--    1 markneedham      wheel               12204 23 Jun 13:44 foo/database-agent-mac490bf-1-logs-archive-201606231244151.tar.gz
121934588        8 -rw-r--r--    1 markneedham      wheel                2895 23 Jun 13:44 foo/database-agent-macbb8e1-1-logs-archive-201606231244151.tar.gz
121934590       32 -rw-r--r--    1 markneedham      wheel               13427 23 Jun 13:44 foo/database-agent-macc8177-1-logs-archive-201606231244152.tar.gz
121934587        8 -rw-r--r--    1 markneedham      wheel                3882 23 Jun 13:44 foo/database-agent-maccd92c-1-logs-archive-201606231244151.tar.gz
121934611        8 -rw-r--r--    1 markneedham      wheel                3970 23 Jun 13:44 foo/database-agent-macdf24f-1-logs-archive-201606231244165.tar.gz

Or we can filter by relative time e.g. to find the files last modified in the last 1 day, 5 hours:

$ find foo -name database-agent* -mtime -1d5h -ls
121939705        8 -rw-r--r--    1 markneedham      wheel                2524 23 Jun 14:00 foo/database-agent-mac17f73-1-logs-archive-201606231300176.tar.gz
121939704        8 -rw-r--r--    1 markneedham      wheel                2511 23 Jun 14:00 foo/database-agent-mac29389-1-logs-archive-201606231300176.tar.gz
121934591       24 -rw-r--r--    1 markneedham      wheel               11294 23 Jun 13:44 foo/database-agent-mac3533f-1-logs-archive-201606231244152.tar.gz
121939707       16 -rw-r--r--    1 markneedham      wheel                4878 23 Jun 14:00 foo/database-agent-mac35563-1-logs-archive-201606231300176.tar.gz
121934612        8 -rw-r--r--    1 markneedham      wheel                3896 23 Jun 13:44 foo/database-agent-mac35f7e-1-logs-archive-201606231244165.tar.gz
121939708       16 -rw-r--r--    1 markneedham      wheel                4887 23 Jun 14:00 foo/database-agent-mac40798-1-logs-archive-201606231300176.tar.gz
121934589       24 -rw-r--r--    1 markneedham      wheel               12204 23 Jun 13:44 foo/database-agent-mac490bf-1-logs-archive-201606231244151.tar.gz
121939720        8 -rw-r--r--    1 markneedham      wheel                2510 23 Jun 14:00 foo/database-agent-mac5f094-1-logs-archive-201606231300189.tar.gz
121939706       16 -rw-r--r--    1 markneedham      wheel                5912 23 Jun 14:00 foo/database-agent-mac636b8-1-logs-archive-201606231300176.tar.gz
121934588        8 -rw-r--r--    1 markneedham      wheel                2895 23 Jun 13:44 foo/database-agent-macbb8e1-1-logs-archive-201606231244151.tar.gz
121934590       32 -rw-r--r--    1 markneedham      wheel               13427 23 Jun 13:44 foo/database-agent-macc8177-1-logs-archive-201606231244152.tar.gz
121934587        8 -rw-r--r--    1 markneedham      wheel                3882 23 Jun 13:44 foo/database-agent-maccd92c-1-logs-archive-201606231244151.tar.gz
121934611        8 -rw-r--r--    1 markneedham      wheel                3970 23 Jun 13:44 foo/database-agent-macdf24f-1-logs-archive-201606231244165.tar.gz

Or the ones modified more than 1 day, 5 hours ago:

$ find foo -name database-agent* -mtime +1d5h -ls
121879391       24 -rw-r--r--    1 markneedham      wheel                8856 23 Jun 11:49 foo/database-agent-mac19b6b-1-logs-archive-201606231049507.tar.gz
121879394       24 -rw-r--r--    1 markneedham      wheel                8772 23 Jun 11:49 foo/database-agent-mac1f427-1-logs-archive-201606231049507.tar.gz
121879390       24 -rw-r--r--    1 markneedham      wheel                9702 23 Jun 11:49 foo/database-agent-mac7e165-1-logs-archive-201606231049507.tar.gz
121879393        8 -rw-r--r--    1 markneedham      wheel                2812 23 Jun 11:49 foo/database-agent-macab7f1-1-logs-archive-201606231049507.tar.gz
121879413        8 -rw-r--r--    1 markneedham      wheel                3144 23 Jun 11:49 foo/database-agent-macbcbe8-1-logs-archive-201606231049520.tar.gz
121879414        8 -rw-r--r--    1 markneedham      wheel                3131 23 Jun 11:49 foo/database-agent-mace075e-1-logs-archive-201606231049520.tar.gz
121879392        8 -rw-r--r--    1 markneedham      wheel                3130 23 Jun 11:49 foo/database-agent-mace8859-1-logs-archive-201606231049507.tar.gz

There are lots of other flags you can pass to find but these ones did exactly what I wanted!

Categories: Programming

Stuff The Internet Says On Scalability For June 24th, 2016

Hey, it's HighScalability time:


A complete and accurate demonstration of the internals of a software system.

 

If you like this sort of Stuff then please support me on Patreon.
  • 79: podcasts for developers; 100 million: daily voice calls made on WhatsApp; 2,000; cars Tesla builds each week; 2078 lbs: weight it takes to burst an exercise ball; 500 million: Instagram users; > 100M: hours watched per day on Netflix; 400 PPM: Antarctica’s CO2 Level; 2.5 PB: New Relic SSD storage; 

  • Quotable Quotes:
    • Alan Kay: The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs.
    • @jaykreps: Actually, yes: distributed systems are hard, but getting 100+ engineers to work productively on one app is harder.
    • @adrianco: All in 2016: Serverless Architecture: AWS Lambda, Codeless Architecture: Mendix, Architectureless Architecture: SaaS
    • @AhmetAlpBalkan: "That's MS SQL Server running on Ubuntu on Docker Swarm on Docker Datacenter on @Azure" @markrussinovich #dockercon
    • Erik Darling: EVERYTHING’S EASY WHEN IT’S WORKING
    • @blueben: Bold claim by @brianl: Most of the best tech of the last 10 years has come out of work at Google. #VelocityConf
    • Joe: there is no such thing as a silver bullet … no magic pixie dust, or magical card, or superfantastic software you can add to a system to make it incredibly faster. Faster, better performing systems require better architecture (physical, algorithmic, etc.). You really cannot hope to throw a metric-ton of machines at a problem and hope that scaling is simple and linear. Because it really never works like that.
    • Eran Hammer: The web is the present and it’s a f*cking mess. Deal with it.
    • @etherealmind: If you believe in DevOps/NetOps you have to believe that leaving Europe is a difficult but better course of action. Small, fast & fluid
    • Sanfilippo: Redis is currently not good for data problems where write safety is very important. One of the main design sacrifices Redis makes in order to provide many good things is data retention. It has best effort consistency and it has a configurable level of write safety, but it’s optimized for use cases where most of the time, you have your data, but in cases of large incidents you can lose a little bit of it. 
    • David Smith: The best time you are ever going to have to make a new app is when there's a new iOS update. Go through the diffs. Go through the What's New? Find something that couldn't be possible before and make an app around that. 
    • @DanielEricLee: There was a timezone bug so I wrote a test and then the test failed because the CI runs in a different timezone and then I became a farmer
    • @jasongorman: Reminds me of someone I know who led a dev team who built something that won an award. None of team invited to awards bash. Only snr mgmt.
    • David Robinson: My advice to graduate students: create public artifacts
    • @cdixon: Because distributed clusters of commodity machines are more powerful.
    • @wmf: 128-port TORs have outrun the compute capacity of most racks, so putting two mini-TORs in 1U is a great idea.
    • msravi: I moved from AWS EC2 to Google Cloud a few days ago. Google really seems to have beaten AWS, at least in pricing and flexibility. On AWS (Singapore region) a 2-vCPU, 7.5G RAM instance costs $143/month (not including IOPS and bandwidth costs), while a similar one on GC works out to about $56/month. That's a massive difference. In addition, GC allows me to customize cores and RAM flexibly to a point, which is important for me.
    • Mobile Breakfast: What is clear that we will get rid of the classic circuit-switched technology and move to all IP networks fairly soon in the US.
    • Douglas Rushkoff: I think they all come down to how you can optimize your business or the economy for flow over growth; for the circulation of money rather than the extraction of money.
    • Alan Hastings~ [Species] that go into synchrony may be more subject to extinction because a single driver can trigger a collapse
    • @TechCrunch: More cars than phones were connected to cell service in Q1 http://tcrn.ch/28MLtmt  by @kristenhg
    • @docker: "Nobody cares about #containers, it's the application that matters!" - @solomonstre, CTO @Docker #DockerCon
    • @cmeik: The most commercially successful NoSQL database: Lotus Notes.
    • Brittany Fleit: behavior-based push results in open rates 800 percent higher than immediate blasts. Personalizing a message based on individual actions garners much more engagement.

  • Dilbert, of course, nails it on AI.

  • How will hackers be stopped from using Return-oriented programming (ROP) to execute privilege escalation attacks? ROP was created when "clever hackers realized that they could successively invoke snippets of code at the end of existing subroutines as 'proxies' to accomplish the work they needed done." Randomizing memory locations didn't stop them so Intel created new hardware magic called Control-flow Enforcement Technology. Intel added a new "ENDBRANCH" instruction and created a new "shadow stack". It turns out the immense power and beauty of the stack in von neumann architectures is also a great weakness. The story of life. Steve Gibson with an inspiring deep dive on CET in Security Now 565

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Beginning An Agile Effort: Breaking Down The Big Picture

26955153324_91f40a31cc_k

How the big picture gets made…

A high-level narrative or story is a good first step in the process of turning an idea into something that delivers real business value.  However, it is only a step along the path through the backlog onion.  Software development (including new development, enhancements, and maintenance work) requires translating the big picture into a product backlog that includes features and epics.  As work progresses the backlog is further decomposed and groomed into user stories and tasks.  The process of moving from any big picture to a backlog in software development follows a predictable pattern.   

  1. Decompose the big picture by identifying the functional and architectural features directly identified or inferred in the story.  This initial step yields a combination of business requirements and the beginning of both the technical and business architecture that will be needed to support the features.  Features of all types are a description of need and outcome and do not represent a decision on how that outcome will be implemented (real options).
  2. Organize the features using a mapping technique such as Agile Story Maps to look for gaps.  Ask yourself and the team whether there are features that needed to achieve the goal identified in the big picture story.  This also a chance for the product owner to reject stories that are not needed to attain the overall goal.
  3. Identify the minimum viable product (MVP). The identification of a minimum viable product at this point identifies the set of features that are needed to deploy a product.  An MVP begins the discussion of priority and a release strategy.  Simply put, features in the MVP will need to be decomposed before features that will be needed later.

An Agile Story Map and an MVP allow a team to cut a wedge into the planning onion that slices from the outer layers directly to the core.  In the Scaled Agile Framework Enterprise (SAFe) the wedge might be called a Product Increment (PI) or a release for efforts using other frameworks and methods.

  1. Features and epics, while easily recognizable, rarely can be tackled by a single team in a single sprint; therefore, they need to be decomposed or split.  The process of decomposing epics and features breaks these large pieces of work into smaller pieces that can be understood and developed within the context of an individual sprint. The backlog of user stories is decomposed as needed so that, as a story gets closer to being accepted into a sprint and developed, the story becomes more and more granular.
  2. Backlog grooming the final step before a story is accepted into a sprint or drawn onto the Kanban board.  Grooming ensures that the story is ready for development (properly formed, sized correctly, understood and have acceptance criteria), and that a team can get started quickly.

The big picture defines the outside of the backlog onion.  However, after we peel the outside of the onion we can cut wedges directly to the core. Slicing wedges, like a minimum viable product, allows a team or teams to decompose and groom only the features and epics immediately needed.  Thereby getting the effort started quickly and effectively.   
Next, a simple example!


Categories: Process Management

Introducing Firebase Authentication

Google Code Blog - Thu, 06/23/2016 - 21:35

Originally posted on Firebase blog

Posted by Laurence Moroney, Developer Advocate and Alfonso Gómez Jordana, Associate Product Manager

For most developers, building an authentication system for your app can feel a lot like paying taxes. They are both relatively hard to understand tasks that you have no choice but doing, and could have big consequences if you get them wrong. No one ever started a company to pay taxes and no one ever built an app just so they could create a great login system. They just seem to be inescapable costs.

But now, you can at least free yourself from the auth tax. With Firebase Authentication, you can outsource your entire authentication system to Firebase so that you can concentrate on building great features for your app. Firebase Authentication makes it easier to get your users signed-in without having to understand the complexities behind implementing your own authentication system. It offers a straightforward getting started experience, optional UX components designed to minimize user friction, and is built on open standards and backed by Google infrastructure.

Implementing Firebase Authentication is relatively fast and easy. From the Firebase console, just choose from the popular login methods that you want to offer (like Facebook, Google, Twitter and email/password) and then add the Firebase SDK to your app. Your app will then be able to connect securely with the real time database, Firebase storage or to your own custom back end. If you have an auth system already, you can use Firebase Authentication as a bridge to other Firebase features.

Firebase Authentication also includes an open source UI library that streamlines building the many auth flows required to give your users a good experience. Password resets, account linking, and login hints that reduce the cognitive load around multiple login choices - they are all pre-built with Firebase Authentication UI. These flows are based on years of UX research optimizing the sign-in and sign-up journeys on Google, Youtube and Android. It includes Smart Lock for Passwords on Android, which has led to significant improvements in sign-in conversion for many apps. And because Firebase UI is open source, the interface is fully customizable so it feels like a completely natural part of your app. If you prefer, you are also free to create your own UI from scratch using our client APIs.

And Firebase Authentication is built around openness and security. It leverages OAuth 2.0 and OpenID Connect, industry standards designed for security, interoperability, and portability. Members of the Firebase Authentication team helped design these protocols and used their expertise to weave in latest security practices like ID tokens, revocable sessions, and native app anti-spoofing measures to make your app easier to use and avoid many common security problems. And code is independently reviewed by the Google Security team and the service is protected in Google’s infrastructure.

Fabulous use Firebase Auth to quickly implement sign-in

Fabulous uses Firebase Authentication to power their login system. Fabulous is a research-based app incubated in Duke University’s Center for Advanced Hindsight. Its goal is to help users to embark on a journey to reset poor habits, replacing them with healthy rituals, with the ultimate goal of improving health and well-being.

The developers of Fabulous wanted to implement an onboarding flow that was easy to use, required minimal updates, and reduced friction with the end user. They wanted an anonymous option so that users could experiment with it before signing up. They also wanted to support multiple login types, and have an option where the user sign-in flow was consistent with the look and feel of the app.

“I was able to implement auth in a single afternoon. I remember that I spent weeks before creating my own solution that I had to update each time the providers changed their API”
- Amine Laadhari, Fabulous CTO.

Malang Studio cut time-to market by months using Firebase Auth

Chu-Day is an application (available on Android and iOS) that helps couples to never forget the dates that matter most to them. It was created by the Korean firm Malang Studio, that develops character-centric, gamified lifestyle applications.

Generally, countdown and anniversary apps do not require users to sign-in, but Malang Studio wanted to make Chu-day special, and differentiate it from others by offering the ability to connect couples so they could jointly countdown to a special anniversary date. This required a sign-in feature, and in order to prevent users from dropping out, Chu-day needed to make the sign-in process seamless.

Malang Studio was able to integrate an onboarding flow in for their apps, using Facebook and Google Sign-in, in one day, without having to worry about server deployment or databases. In addition, Malang Studio has also been taking advantage of the Firebase User Management Console, which helped them develop and test their sign-in implementation as well as manage their users:

“Firebase Authentication required minimum configuration so implementing social account signup was easy and fast. User management feature provided in the console was excellent and we could easily implement our user auth system.”
- Marc Yeongho Kim, CEO / Founder from Malang Studio

For more about Firebase Authentication, visit the developers site and watch our I/O 2016 session, “Best practices for a great sign-in experience.”

Categories: Programming